Aflevering 1: Het fantoom-plan

If you are an RIA or looking to become one, Schwab advisor services is the Watson to your homes. Smart, dependable, always has your back. It's no mystery why they're the number one choice for RIAs who want to be the ultimate solution for their clients. With all the wealth services, technology and support your firm needs, the difference is so obvious. It's Schwabbius. Follow the clues at Schwabbius.com. Your fabric bill keeps climbing, but your data volume barely moved. So teams do what teams always do. They blame growth, they blame licensing, they blame the SKU. But in a lot of environments, that's not the problem. The bill isn't reacting to how much data you stored. It's reacting to how your workloads behave every minute of the day. And that's where this breaks. Because fabric puts shared compute behind reports, refreshes, SQL pipelines, notebooks, warehouses, semantic models, all of it. So low value work doesn't stay local. It competes with the work that business actually cares about. In this episode, I'm going to show you where capacity units really go, why those spikes keep coming back, and what you can fix in the next 30 days. If fabric cost and governance matter to you, follow this podcast because this is exactly the kind of problem we unpack here. The model behind the bill. To understand the bill, you need to understand the model behind it. Fabric doesn't charge you like a stack of separate products. It gives you one shared capacity pool, measured in capacity units, and that pool gets consumed by reports, refresh jobs, SQL endpoint queries, pipelines, notebooks, warehouses, lakehouses, semantic models, and more. That shared model is the promise. If one workload is quiet, another workload can use the free headroom. On paper, that's efficient. And sometimes it is. If your usage is disciplined, shared compute works well because idle capacity doesn't just sit there waiting for one product team to wake up. It can be reused. That's the upside. But the moment teams start treating every engine as the default path or the scheduled background work with no coordination, the same design turns against you, and one team's convenience starts showing up on everybody else's invoice. That's the shift most teams miss. They still think in silos. One report team, one data team, one SQL team, one pipeline owner. But Fabric doesn't bill in silos. It builds the shared pool. So a badly timed refresh, an expensive SQL pattern, or a noisy export process doesn't just hurt that one workload. It takes capacity away from everything else running in the same place. Now add one more layer because this is where people usually misread the charts. Fabric separates interactive operations from background operations. Active means user-driven activity, things like report queries, background means scheduled or system-driven work, refreshes, pipelines, data processing. And in a healthy environment, that split helps you see what users are asking for versus what the platform is doing behind the scenes. But background work can quietly eat your headroom before users even arrive. So when the business opens dashboards at 9 o'clock, the capacity already enters the day tired. Then smoothing makes the picture even less obvious. Short operations can be smoothed over at least five minutes, while long background operations can be smoothed over up to 24 hours. That means a short spike doesn't always look short on the financial side. Fabric also tracks carry forward overage. So excess usage can keep affecting the capacity after the original burst finished. That's why a team can say, "The spike only lasted a few minutes." And still feel pain much longer through delays, rejections, or extra cost. So when people obsess over storage growth, they're watching the wrong thing. Storage matters sure. But the invoice reacts far more to compute behavior, query design, refresh overlap, and concurrency. You're not mainly paying for bytes at rest. You're paying for thousands of compute decisions, spread across reports, jobs, and endpoints all day long. Once that shared model clicks, the next question gets very practical. Where is the money actually going? Where CUS actually go. So where did the CUS actually go? Start with the metric that changes the conversation fastest. CUS consumption by item. Not monthly invoice total. Not workspace size. Not how many users you think are active, item level consumption. Because that's where the pattern stops being abstract and starts naming names. Open the Capacity Metrics app and look at the top consumers. In a lot of environments, you won't find a broad, evenly distributed load. You'll find concentration. A small set of assets pulling a huge share of compute sometimes 10 items explain most of the pressure. And once you see that, the story shifts from fabric is expensive to these specific things are expensive. That's a very different problem. In most organizations, the top offenders are pretty predictable. SQL endpoint queries, semantic model refreshes, data flows, pipelines that overlap more than anyone realized. And sometimes exports or ad hoc access patterns that look harmless because each action feels small, but they repeat all day and keep pulling from the same pool. So what looks like normal usage from each team turns into constant pressure at the capacity level. The thing most people miss is the split between interactive and background activity. If the interactive side is high, that usually maps to actual user demand. Reports opening, filters changing. Usually running because people are using the platform that may be fine. But when background usage dominates, especially before the business day even starts, you need to ask a harder question, is this worth creating value or is it just running because nobody challenged the schedule? That distinction matters because user demand and system waste don't get fixed the same way. And this is where time of day becomes useful. Don't just ask what consumers you use. Ask when. Weekday peaks tell one story. Quarter end reporting tells another. Repeated surges every 30 to 60 minutes usually point somewhere very specific. Press overlap, pipeline collisions, recurring jobs or query bursts tied to the same operational rhythm. Once you line up the time pattern with the top items, the hidden structure starts to show itself. You'll often see a pattern like this, calm periods overnight, then scheduled background jobs kick in, capacity climbs before users arrive. Business hours begin, interactive demand lands on top of unfinished background work and suddenly performance drops. Not because one dashboard is broken, but because the room was already crowded before the meeting started. But also why the throttling events timeline matters so much. If throttling appears as random noise, maybe you're dealing with genuine variability. But if it shows up in repeatable spikes, same times, same windows, same kinds of items, you're looking at a design issue, usually concurrency, sometimes rooting, often both. A raw under provisioning problem would look more like steady pressure. These environments don't, they pulse. And fabrics own guidance helps here. When peak utilization keeps pushing into the 70 to 80% range, you need to act. Below 40% may point the other way, bad allocation, over sizing or capacity sitting underused. So the job is not keep utilization low at all costs. The job is to understand whether the shape of utilization matches business value. High usage during real demand can be healthy. High usage from low value repetition is just waste with a chart attached. If you remember nothing else from this section, remember this. Don't start with the bill. Start with the top items, the interactive versus background split, the time pattern, and the throttling timeline. Those four views usually tell you who is burning the budget, whether users caused it, and whether the issue is constant demand or preventable collisions. Once you have that, the next question gets sharper. Not who consume the most, but why those items consume so much in the first place. Scenario one, the SQL convenience tax. One pattern shows up again and again in client environments. And it usually starts with a reasonable decision. People know SQL, analysts know SQL, engineers know SQL, report builders trust SQL because it feels direct, familiar, and fast to start with. So the SQL endpoint becomes the easiest answer to almost every request. Need a quick query use SQL, need an export use SQL, need a report source, use SQL, need a shortcut because the business wants something by Friday, use SQL again. That's not a technical failure, it's a behavior pattern. And once that pattern settles in, SQL stops being one access path and turns into the default compute layer for the whole platform. That's where cost starts drifting away from value. I've seen environments where SQL workloads took roughly 70 to 80% of total CU consumption, while the business value behind those queries was all over the place. Some useful, some repetitive, some just habits nobody challenged because the endpoint kept saying yes. The line I keep coming back to is simple. SQL was not wrong, it was used for everything, that distinction matters. Because teams often defend this setup by saying, but SQL is simpler. And they're right from the user side, simpler to write, simpler to hand over, simpler to debug if your team grew up in that world. But simple for the user can be expensive for the platform, especially when the same endpoint gets hit by broad scans, repeated reads over large delta tables, ad hoc exploration, exports and workloads that really belong somewhere else. This clicked for me when I saw how often the endpoint had become a universal answer, not because it was the best engine for each job, but because it removed the need to decide. And once you stop deciding, you stop routing work properly. Research makes that contrast hard to ignore. Spark direct operations can be far more efficient than rooting the wrong job through a SQL endpoint, and one benchmark in the source material is brutal. A complex query pattern through a spark to SQL endpoint connector came out 32 times costlier and 73 times slower than running directly in Spark. That's not a small tuning gap, that's the platform charging you for architectural laziness. Now that doesn't mean SQL endpoints are bad, it means they fit some jobs and punish others. Lightweight reads, familiar access, straightforward consumption patterns, fine. Heavy transformation, repeated scanning, using connectors that stack one engine on top of another, that's where the bill starts swelling without a matching jump in business outcome. But typically happens next is predictable. During business hours, the SQL load climbs because everyone is doing the convenient thing at the same time. Interactive pressure rises, other workloads lose room, and then Spark heavy work starts getting throttled or delayed. The operators look at fabric and think the platform is unstable, but in many of these cases, the platform is doing exactly what the workload makes for us to do. So the real fix is not to ban SQL. That would be silly. The fix is to stop treating SQL as the universal front door. You need a routing decision, which jobs belong on SQL because they're light and read focused, and which belong on Spark because they transform, scan hard, or scale badly through endpoint patterns. Without that decision, convenience keeps winning, and convenience has a CU price. And once you see that first leak clearly, another one usually appears right next to it, not from familiarity this time, but from teams scheduling work with no shared control at all. Scenario 2, the invisible refresh storm. The second pattern looks different on the surface, but it creates the same outcome. to get strained by work that feels like it. is normal to each team in isolation, while nobody sees the combined effect until users start complaining. This time, the driver isn't convenient. It's coordination failure. What typically happens is simple. The BI team schedules data set refreshes based on report needs. A data team schedules data flows on its own cadence. Someone else adds semantic model refreshes. Pipelines run when upstream jobs finish or when someone guessed they should. Each choice sounds reasonable. Each team has a local reason. But nobody owns the full refresh map across the capacity, so the system fills up with overlapping background work that keeps colliding. Then the pattern shows up in the metrics, spikes every 30 to 60 minutes, the same windows, the same bursts, concurrency rises, throttling events appear, and report users start feeling lag right when the business needs stable performance. From the front end, it looks like fabrics slow down. One layer deeper, the platform is busy replaying a schedule nobody designed as a whole. The line that usually explains the mess is this. No one owned refresh orchestration, so everyone scheduled independently. That's not a small operational detail. It changes the economics of the platform. Semantic model refreshes can need much more memory than the final model size because new structures are built while the old version is still in memory. Research in the source material points out that refresh operations can require two times or more memory than the resting model. So a model that looks manageable at rest can still create a large processing peak when it refreshes. Now stack several of those peaks together, add data flows, add pipelines, and you've built pressure that was never visible in storage numbers. And there's another reason this stays hidden for too long. Background work doesn't always fail loudly. It just keeps consuming headroom. Fabric smooths longer background operations over extended windows, which means the load can linger financially and operationally after the trigger event passed. So the people who scheduled the jobs think they ran overnight or every half hour and finished. The users who open reports later feel the consequence. Those two experiences don't look connected unless someone studies the capacity behavior as one system. This is also where the fabric is under provisioned argument starts to fall apart. If the environment truly lacked enough capacity all day long, the graphs would show broad sustained pressure. But that's not what these cases look like. They jump. They repeat. They cluster around windows that match refresh cycles and pipeline timing. That's not a pure capacity story. It's the concurrency story. The same goes for data volume is the problem. Data growth matters, but it doesn't explain why modest changes in stored data can sit next to much larger jumps in CU consumption. What explains that gap is execution getting worse, more full refreshes, more overlap, more duplicate processing, more schedules layered on top of each other because every team solved freshness on its own. So the fix is not another dashboard showing the pain after it happens. The fix is governance that changes behavior before the next collision starts. Someone has to own the refresh calendar, the dependency logic and the order of operations across teams. Without that, background work keeps looking invisible right up until the moment the business feels it. Why more capacity usually makes this worse? So when teams finally see the pressure, the first instinct is obvious. By more capacity, move up the SKU, turn on overage, give the system more room and hope the problem settles down. And for a little while it often does. Dashboards open faster, complaints drop. The graphs look calmer. Leadership feels relief. But the model behind that relief is dangerous because extra headroom can hide bad behavior instead of correcting it. If low value queries, poor rooting, and unmanaged refresh overlaps stay in place, new capacity doesn't remove the waste. It gives the waste more space to grow. The expensive query still runs. The overlapping refreshes still collide. The end point still gets used as the default answer. You just stop feeling the pain immediately, which makes the organization think the design is now fine. It isn't. An overage has the same trap. It can help with real bursts. That matters. If you have quarter end demand, planned reporting peaks or known windows where you genuinely need flexibility, overage can protect the business from hard throttling. But if overage becomes normal, then you didn't solve variability. You funded it. At a higher rate. In the research, overage is positioned as a safety net, not an operating model. The moment it becomes your routine answer, you're paying premium rates to preserve weak governance. That's why I push back when people say we need more capacity to scale. Maybe. Sometimes they do, but scale has shape. If the pressure is broad, stable, and tied to real demand, more capacity may be the right move. If the pressure is clustered, repetitive, and tied to the same collisions every day, more capacity just lets broken routing and bad schedules keep expanding. You're not scaling growth. You're scaling waste. And one level deeper, this isn't really a tooling problem. It's an ownership problem. Who decides which engine gets used for which kind of work? Who decides when refreshes run? Who stops one team from pushing cost into a shared pool? Everyone else depends on. In a lot of fabric environments, the answer is nobody with enough authority to enforce it. So the platform becomes permissive. And permissive systems always drift toward higher spend. That matters for executives because this doesn't show up as one big bad decision. It shows up as hundreds of small decisions that look harmless on their own. One more refresh, one more export, one more query patent nobody challenged. One more workload left on the wrong path because moving it felt inconvenient. Add those together across a shared capacity and the bill climbs long before anyone admits the operating model is weak. So the message to leadership is pretty blunt. You are not paying for data. You are paying for compute choices repeated all day across teams that may not even realize they are affecting each other. Until those choices get governed, capacity will keep feeling smaller than it should, no matter how much you buy. That's why the right next move isn't panic purchasing. It's a short governance reset. The 30 day governance reset. If you want the bill to calm down, you need a reset that changes workload behavior fast. Not a six month architecture program, not a giant redesign, 30 days. Clear owners, clear rules, and a weekly review rhythm that forces choices. Start with workload isolation. Split BI and SQL heavy demand from engineering and spark heavy work where that pressure is hurting each other. That can mean dedicated capacities or a cleaner separation of workspaces and critical workloads, so one side stops draining the other. The point is simple. Stop letting business facing consumption compete directly with heavy engineering patterns when they don't need to share the same blast radius. Then put guardrails around the SQL endpoint, not because SQL is bad, but because open-ended convenience becomes expensive fast. Look for long running queries, broad scans, export heavy patterns, and repeated access against large tables where the endpoint keeps getting used as the default answer. You need standards for what belongs there, what doesn't, and who gets to approve exceptions. If nobody owns that boundary, the endpoint keeps expanding until it becomes the bill. The next move is refresh orchestration. This one usually pays back quickly because the waste is often mechanical. Centralized scheduling, stagger refresh windows. Remove duplicate refreshes that exist only because teams built around each other without knowing it. Where semantic model refreshes can collide, use dependency-aware pipelines and waiting logic, so one process checks whether another is still running before it starts. Research around fabric refresh conflicts points to polling patterns as a practical fix, because retries alone don't solve unpredictable overlap. After that, make cost visible by team, domain, and use case. Showback changes behavior because it removes the shared illusion. Once teams can see which workloads consume the capacity, the conversation gets more honest. The goal isn't to shame anybody. It's to stop treating the capacity like free air. Fabrics charge back and monitoring capabilities exist for exactly this reason. The moment people see who burns what, optimization stops feeling optional. Then define a rooting strategy. Heavy transformation work goes to spark. Lightweight read patterns can stay where SQL fits. And that sounds obvious, but most environments drift because nobody writes the rule down. So write it down, which query shapes stay on SQL? Which data preparation parts go elsewhere? Which shortcuts are allowed in dev, but not in production? Governance works when the model is visible enough that teams don't have to guess every time. And don't turn this into a dashboard hobby, put an operating rhythm behind it. Every week review the top CU offenders by item. Look at the throttling pattern. Ask one decision for each top consumer. Optimize it, isolate it, or retire it. That's it. Small set of assets, clear action, repeated weekly. Because the fastest path to control is rarely platform wide reform. It's direct pressure on the few things driving most of the pain. That matters for leadership too. Executives don't need every technical detail, but they do need the model. A small number of assets can drive most of the cost, so governance should start there, not with broad policy language no one can enforce. Pick the top offenders, assign owners, set deadlines, review again in seven days. When teams know the bill is being traced back to workload choices, the whole platform starts behaving differently. And this is the shift. Stop thinking about fabric as a storage problem with a bigger invoice attached. Start treating it as a shared compute system that needs routing, ownership and discipline. So the issue isn't mainly how much data you keep, it's how shared compute gets consumed by which workloads at what times and with what level of control. Pull your top CU by item view, take the top 10 offenders and label each one, keep, fix, isolate, or retire. If this changed how you think about your fabric bill, leave a review, it helps more teams find the show and connect with me, Mercopeters, on LinkedIn, with the next cost pattern you want unpacked.