No AI without data: Why digital success starts with the basics

Welcome to The Deep Dive. We take a pile of research, connect the docs, and hopefully give you the shortcut to understanding what's really going on. And today, we're diving into artificial intelligence, you know, if you follow any business news, the hype is just immense. We keep hearing about these amazing recommendation engines, perfect pricing, automated everything. Yeah, AI is pitched as the next great leap forward. And look, in many ways, it has that potential. I sense a butt coming. Well, yeah, if you look at the numbers, the reality on the ground that our sources point to that leap often falls short, way short. Okay. We found this really sobering truth almost immediately. Most AI projects actually fail. And the kicker is it's usually not the algorithms, not the fancy math. So what is it then? It's the data, the stuff those algorithms are supposed to learn from. It's just not good enough. Okay, let's put some numbers on that because a couple of these stats really jumped out at me. One analysis suggests something like 85% of AI and machine learning projects just fail to deliver what they promised. 85%. It's a shocking number, isn't it? I mean, imagine investing that much in anything else with that kind of failure rate. Unthinkable. And the cost, what one study we looked at estimated the cost of poor data quality just in the US was around 3.1 trillion dollars a year. That was back in 2016, mind you. Yeah, 3.1 trillion. It sounds almost unbelievable, right? But it points to this deep underlying problem in how companies handle their information. It's not trivial. Absolutely not. So our mission today is to unpack why? Why are companies even ones with tons and tons of data struggling so much? What does bad data actually look like day to day? And crucially, what's the essential, maybe unglamorous work that has to happen first before you even think about deploying sophisticated AI? Right. Let's start with this idea. You flagged the curse of the data gold mine. Sounds paradoxical. It kind of is think about distributors or e-commerce companies manufacturers. They seem perfectly set up for AI, don't they? You think so. They track everything. Exactly. They're sitting on mountains of data. You know, scoyoo level details, transaction histories, pricing moves, every customer click, every quote, every return. It looks like a gold mine. All the raw materials seems to be there. But our sources were pretty clear, just having a lot of data isn't the same as having good data or ready data. Not even close. Yeah. That abundance is really deceptive. The data is almost never clean. It's often siloed, you know, marketing has its data, sales has theirs, finance has there, and they don't talk to each other. Right. Different systems, different formats. Totally. And it's unstructured, it's outdated, it's inconsistent. Let's make that real though. What does that actually mean? Yeah, give us some of those examples, what are the AI teams pulling their hair out over? Okay, think about your product data, your SKUs. The descriptions might be incomplete or just plain wrong. Different teams might use different codes for the same thing. Then you got customer records. Duplicates all over the place. So your personalization engine thinks one person is three different people. Or pricing data, maybe it doesn't show why a certain price was given, was it a promotion, a manual override, who knows? And sales data must be a minefield. Oh, yeah, riddled with manual overrides, exceptions, side deals that don't follow any clear logic the AI can actually learn from. It's just noise. So the company invests in this powerful AI feeds it this, well, this messy history. And the AI doesn't magically fix it. It actually makes it worse. It amplifies the flaws. Amplifies how so. Well, AI looks for patterns, right? If the patterns in your data are based on years of, let's be honest, neglect and inconsistency, the AI just learns those bad patterns, reinforces the chaos. So the data that was supposed to be an asset becomes this huge liability. Precisely. It just shines a spotlight on years of maybe not paying enough attention to data hygiene. Which brings us to that staggering cost, that $3.1 trillion figure. The sources talk about something called the hidden data factory. Explain that. Right, the hidden data factory is basically all the unproductive work people do because the data is bad. It's the time and money spent by smart, expensive people, analysts, managers, decision makers, just trying to work around the flaws. Fixing things manually, cross checking reports. Exactly. Hunting for data they actually trust. Correcting errors from other departments, trying to reconcile conflicting numbers. It's all non-value added work. And the time sync is just incredible. One step said knowledge workers waste up to 50% of their time on this stuff. Half their day. Half their day. Think about that. And the salary is spent essentially compensating for bad data, not on driving the business forward, not innovating. And the data scientists, the people you hired specifically to build the AI models. They're often the worst hit. Estimates suggest they spend maybe 60% of their time just cleaning and organizing data. Before they can even start the actual analysis or modeling. 60% adds to your hidden factory right there. That's a really powerful way to frame it. It's not some abstract IT problem. It's a direct operational cost. Department B is literally spending time and money fixing mistakes. Department A made upstream. And the customer isn't paying for that internal cleanup. That inefficiency, scale across the economy, that's where that multi-trillion dollar cost comes from. If the data was right the first time, well the factory wouldn't need to exist. Makes perfect sense. So, okay, we see the cost of bad data. Now, let's connect it directly to why the AI models themselves break down. This explains that 85% failure rate. Right. The core principle is simple. Almost a cliche, but it's true. AI applications are only as good as the data they're trained on. Garbage in, garbage out basically. But the sources get really specific about how different types of data flaws cause different types of AI failures. Well, off into those technical breakdowns. Okay. One of the big ones is overfitting. This is where the model learns the training data too well. Including all the noise, all the errors, all the weird exceptions. It memorizes the messy past instead of learning general rules. Exactly. It fits the historical data perfectly. But then when you show it new, real-world data it hasn't seen before, it completely fall apart because it didn't learn the underlying signal just the noise. Why? What else? Then there's edge case neglect. Your training data might not have enough examples of rare but really important events. So, the model just doesn't know how to handle them when they happen. Think fraud detection or safety systems. Missing edge cases can be critical. We also see huge problems with correlation dependency. The AI finds patterns, sees that two things often happen together in the historical data, and assumes one causes the other. When it might just be a coincidence or they're both caused by something else entirely. Right. So the model makes recommendations or predictions based on these superficial maybe totally wrong assumptions. Which sounds like a recipe for embedding old biases or just bad habits into your automated systems. Absolutely. Which leads directly to data bias. If your historical data reflects past discriminatory practices, or if certain groups or citations are underrepresented, well, the AI will learn those biases and perpetuate them. Sometimes even amplify them. A huge ethical and practical problem. Massive. And finally, there's data drift. The world changes, right? Customer behavior shifts, market conditions change. If the AI model isn't designed to adapt or be retrained on fresh data, its performance will degrade over time simply because the reality was trained on doesn't exist anymore. So all these factors combined. It makes that 85% failure rate, and the average cost of $12.9 million per year per organization for these failures seem almost inevitable if you skip the foundational data work. It really does. You just can't build reliable AI on shaky foundations. It highlights that the real digital transformation work isn't always the sexy algorithm development. It's the data discipline, treating data like critical infrastructure, not just exhaust fumes from transactions. Exactly. That has to be the starting point. Okay. So if the problem is clear, stop building AI on sand. Let's talk solutions. What does this foundational, maybe difficult data work actually involve? The sources say it's often a multi-quarter cross-functional slog. It absolutely is. It's not quick. And it requires buy-in from pricing, sales operations, IT, category managers, finance. Pretty much everyone who touches data. So where you start? It usually kicks off with a detailed audit. Seriously detailed. You need to map out where all the critical data actually lives, which systems, who owns it, how is it currently used, and importantly, where are the gaps? And this has to include the unofficial systems too, right? The spreadsheets. Oh, absolutely. You have to tackle the shadow systems. Those complex spreadsheets maybe access databases, people built on the side. Yeah. That's often where the real business logic and the clean-up data actually lives, because people couldn't trust the main systems. You need to understand what they're doing and why. How do you get people to give up their carefully crafted spreadsheets? That sounds politically tricky. It can be. The key, according to the sources, is not to frame it as taking something away, but as building something better together. Show them how a centralized clean system eliminates the reason they needed the spreadsheet. It reduces that hidden data factory work for everyone. Their work around often becomes the requirement for the new system. Smart. Use their expertise. Okay, after the audit. Then comes the big clean-up. Product data rationalization. This means getting product descriptions consistent, applying standard categories, making sure units of measure are aligned everywhere. You might even need to rethink old product hierarchies that don't match how customers buy today. A lot of detailed work. Yep. Then, similarly, customer data cleanup. Finding emerging duplicate accounts, figuring out parent-child relationships for big companies, linking buying history correctly. For major accounts, this often needs manual review. You can't just automate guessing which ABC Corp is the right one. And for things like pricing AI, getting the transaction history right is key. Absolutely vital. That's step four. Structuring your transactional data. You need clear rules and systems for capturing why a price was given was at standard A promotion. A negotiated override. The AI needs to distinguish between a deliberate pricing strategy and just noise or an error. So you need guardrails on how overrides and exceptions are recorded. Okay, so you've audited. You've cleaned products, customers, transactions. What's next to make it stick? Now you need the infrastructure and processes for the long haul. Data integration is key. Bringing data from all those different sources, your ERP, your TMS for transportation, your WMS for warehouse operations into one unified view. Using tools like ETL or ELT. Exactly, extract, transform load or extract load transform. Different ways to get data cleaned, standardized, and into a data warehouse or lake where it can actually be used effectively. And then ongoing maintenance presumably? Data governance? Yes, that's the final piece. On going data quality management, DQM. This means setting up clear data governance standards, roles, responsibilities. Who owns which data element? What are the quality rules? You also need to account for regulations like high-palley, for health data, or GDPR for personal data privacy? And embedding it in workflows. Crucially. Good data practices need to become part of the daily routine. So for example, when a salesperson overrides a price, the system prompts them to categorize why. That override isn't an error anymore. It becomes a valuable structured data point for the AI to learn from next time. This really paints a picture of what digital readiness truly means. It's less about the flashy front end tools, and more about the hard, often invisible work of getting the underlying data house in order. That's the core message from all the sources, really. AI is fundamentally math applied to data. If the data isn't trustworthy, isn't structured, isn't ready? The math can't help you. Investing heavily in AI before you fix the data. It's like building that skyscraper on sand. You're just setting yourself up for failure, expensive failure. But the flip side is getting the data right has huge benefits beyond just enabling AI, doesn't it? So absolutely. Reducing that hidden data factory cost alone is massive. Potentially cutting that wasted effort by 66% or more. Some sources suggest clean, reliable data is the gift that keeps on giving. It enables everything else, smarter pricing, yeah. But also smarter inventory management, more effective sales strategies, better customer service, everything. Right. It lifts all boats. Exactly. So we've seen pretty clearly that AI fails mostly because of bad data, and that fixing that data is this intensive cross-functional effort. It takes real discipline. Which leads to a pretty pointed final thought. Yeah, provocative question for you, the listener, to be chew on. If getting data right requires this sustained, multi-quarter commitment across departments that maybe don't always work well together, sales, finance, IT ops, what really needs to change permanently. You mean organizationally? Yeah, what organizational structure or maybe what incentive system needs to be put in place so that prioritizing data quality isn't just a one-off project, but becomes more important than, say, just rushing the next shiny AI pilot out the door. How do you make data first the default culture? That's the million dollar or maybe trillion dollar question, isn't it? How do you hardwire that discipline into the business? That's the challenge. And figuring that out might be what really separates the long-term AI winners winners from the ones who just keep hitting that 85% failure rate.