Unknown

I'm Shail Khan. I lead early-stage investments at Energy Impact Partners. Welcome to Catalyst. So of all the fights over data centers, I think water might be the one where the two sides sort of talk past each other the most. Opponents say data centers are draining aquifers and competing with towns for drinking water. Proponents basically say it's a nothing burger. It's a rounding error next to agriculture and other major water consumers. And actually, new data centers barely use any water at all. I think there are two things going on here. The first is about context. So say a 300-megawatt data center with evaporative cooling uses like 150 million gallons of water per year. Is that a big number or a small number? Just for context, a combined cycle power plant of the same capacity would use roughly three times as much water. Or actually, maybe even better, 150 million gallons per year is about three times as much water per year. The second, though, is about history versus future. I think even the data center industry would admit that water consumption was significantly higher in the past. But I'll also tell you that modern designs, which might be closed-loop, consume very little water at all, perhaps down to essentially just restrooms and kitchens. So rather than trying to argue the numbers, I wanted to understand where the water actually goes. What inside a data center needs cooling? What is the challenge there? How did we do it historically? And what's changed now that a single rack can draw what used to power a whole row? It turns out that the answer to how much water the data center consumes is actually mostly a design choice. But it's a design choice being made right now at gigawatt scale, and the old numbers don't actually tell you much about the way it's going. So to walk through it, I talked to Tim Shedd, the founder of Abolient Insights and an expert in data center cooling. That's coming up next. Tim Shedd: I'm Tim Shedd, and I'm the founder of Abolient Insights and an expert in data center cooling. And today, I'm going to talk to Tim Shedd about how data center cooling works and how it's designed. So today, I'm going to talk to Tim Shedd about how data center cooling works and how it's designed. I'm Tim Shedd, and I'm the founder of Abolient Insights and an expert in data center cooling. Tim Shedd: So today, I'm going to talk to Tim Shedd about how data center cooling works and how it's designed. So today, I'm going to talk to Tim Shedd about how data center cooling works and how it's to every server for every watt that i'm trying to cool and at some point that kind of breaks down um at least from that model so what we did to start to try to make things more efficient as we went from nine kilowatts to some high performance computing going to 30 kilowatts and 40 kilowatts a rack is put an in-row cooler next to it so we bring the cooling closer to the heat source that saves me a lot of energy actually a rear door heat exchanger or an in-row cooler they're have kind of the same basic concept of just bringing that heat exchanger as close as possible to the heat source that way my fans are working very efficiently because i'm only moving heat really closely to my compute and i'm only using fan power that i need and if the racks next to or attached to the radar heat exchanger don't need to be cooled the fans dial back and i'm saving a lot of energy actually but that requires plumbing now onto the the data center floor uh made some people uncomfortable but overall you know it's a good idea but even that gets limited at about 80 kilowatts we've you know i was working for dell and we shipped racks regularly at up to 80 kilowatts a rack with rear door heat exchangers but beyond that it gets difficult to pull enough air through to cool so that's what brings us to today it's not you know radar heat exchangers are actually really efficient and um in many cases almost as efficient as direct liquid cooling up to that power amount but beyond that it just starts to get really difficult okay so to bring it forward to today then you said you know you were shipping 80 kilowatt racks at dell where are we today like what are the in in a new state-of-the-art frontier inference or training data center like what's the rack density everybody probably listening to this podcast is aware of the virarubin mvl 72 or vr mvl 72 racks those are rated at 220 kilowatts a rack including the power supplies and everything else so um that's a lot of power in iraq but that's actually not the densest racks that are shipping today um as an example um dell is installing the uh dudna um super computing cluster at lawrence berkeley national labs and and that power level kept creeping up i want to say it's about 280 kilowatts in iraq um but i i don't remember that exactly but my point is that that 220 kilowatts is actually not the limit today um there there are ways to keep packing more and more uh gpus into a a rack um even today and um whether they're 100 percent liquid cooled like the mvl 72 or they're a mix of air cooling and liquid cooling like the um the dude in the supercomputer um we can still manage that level in fact we can manage we've we've mapped out up to 400 plus kilowatts um 500 kilowatts in that rack so um that is what's possible today uh from a cooling perspective and from a power delivery perspective using orv3 uh racks and so on so yeah we should expect to see that type of density uh continuing on right presumption would be that and density continues to increase that's the direction of travel and regardless it's a you know to your point like we could do 80 kilowatts air cooled maybe we could squeak out a little bit more than that regardless we're multiple of that now so clearly we are forced into doing something other than air cooling which is liquid cooling so walk me through the mechanism what is what is liquid cooling actually look like in iraq and then and then we should talk about the water side of it ultimately what we have to do is bring water to every device that needs to be cooled with liquid so the reason i qualify that is there are different architectures for servers today some are like the the eight-way uh dgx uh architectures today those are still typically a hybrid cooled solution so the gpus and and mv link and other devices are cooled with water but storage might still be air cooled network cards might still be air cooled and so on so i'm going to have some mix of airflow through the server as well as as liquid and in other cases like the mbl 72 it's 100 liquid cooled so i'm bringing water into the server and having to route it to every device somehow touching every device greater than about a half a watt so it's a really intense process once we decide that everything's going to be liquid cooled so imagine just it's a plumbing problem i've got two ports and then a supply and a return on the server and i've got to route that liquid to every cold plate to every piece of copper basically with enhanced you know with fins on it that i'm going to run the water through to pull the heat off of of all the hot devices in addition we typically want to make this serviceable so in other words we want to make it so that we can remove one gpu out of eight or four that are in there that means that all those liquid connections have to have some sort of connector that allows us to remove the gpu without spilling liquid all over the place so in addition to the plumbing to just connect all the copper together that we're using to cool the the the processors we also have manifolds and we have quick connects and we have hoses and other things so there's it's the the liquid cooling infrastructure is actually quite significant and takes up a significant volume within the server today so um as far as the the cold plates themselves they are essentially um just think of it as a as a basically a piece of two million and a half pounds of liquid cooling infrastructure of two millimeter thick copper maybe it's one one and a half millimeters depending and then we've um generally created uh people call them different things but skived fins or fins that are made by shaving a bit of copper off the surface and flipping it up um and uh the typical spacing is about 100 microns thick for the copper and about 100 microns spacing between um i in historically we call those micro channels micro channel cooling today has a different connotation we can dive into that if we want but um they're typically about three millimeters tall and we're going to try to arrange them such that we're flowing water through every one of those micro channels and pulling heat off and that's non-trivial but if we do so they're very high performance we can now cool a you know 1400 watt 1800 watt processor using 1400 or 1800 watt processors that are very high performance and we can now cool a you know 1400 watt 1800 watt processor and uh that's non-trivial but if we do so they're very high performance we can now cool a you know 1400 watt 1800 watt processors using 45 degree c water that's phenomenal that's actually something we couldn't do three years ago so a lot of progress has been made um and then a lot of additional technology and figuring out how to seal those things so they don't you know they don't leak easily they can be shipped across the oceans and and not burst or have any other problems there's a lot a lot of engineering has gone into the cold plates in a pretty short time to make them reliable and high performance can you orient me a little bit on there's different architectures within liquid cooling and you hear these different terms just give me like the the sense of direct-to-chip cooling immersion cooling people talk about two-phase like what how should i think about categorizing the different ways to do liquid cooling uh direct-to-chip means bringing the coolant to the chip and so i structure chip and that can be either single phase or two phase um so um the most the dominant form today is single phase direct-to-chip cooling using water with 25 propylene glycol mixed in so it's basically um crudely speaking it's basically the antifreeze that you pour into your car diluted to 25 when you go buy it at the um auto parts store it's often 50 next premix the really important detail that you don't even think about when you go pick up a bottle of antifreeze is that that you have a very small percentage about one percent to two percent maybe of a proprietary chemical mix it's called an inhibitor or inhibitor package that prevents corrosion so your car has aluminum and brass and all sorts of other materials in it and lasts for years without corroding and falling apart that's actually phenomenal that's amazing so our partners at you know like valvoline and dao and ricocam and they've done an amazing job developing these these coolants that are very long lasting and allow us to use many different materials in in their systems that's the same thing we have to pay attention to in the single phase liquid cooling system so um that's single phase liquid cooling two-phase liquid cooling is not yet really available at scale there are companies working on it i myself had a company that worked on this um so i'm a little you know hesitant to say that i'm a big fan of this um so i'm a little you know have some some history with this um but there are definitely um products that are very close to commercialization in this space and it's where you take a refrigerant you pump it to the chip and you literally boil the refrigerant off of the chips uh you know off of the fins um it looks really cool bubbles are great uh it turns out that that the performance is very similar to single-phase. It's not really superior in performance, but there are other benefits. I can use much smaller hoses and tubes and so on. So remember I was saying currently the DLC takes up a lot of real estate inside of the servers. Two-phase promises, again, not mass-adopted yet, but promises to use a lot less space inside of the servers, leaving more space for innovation on the actual compute. Immersion can take two forms, in single-phase and two-phase. In single-phase, we use an immersion fluid, a coolant, that is often derived from something like mineral oil, but it's generally much more sophisticated than this, and has additives and other inhibitors as well. The problem with oil is if you can imagine it's a hot day and you're getting excited, you jump in the pool and that pool is filled with oil, that would feel pretty not great. It wouldn't cool you off, right? Because oil is actually itself a pretty poor heat transfer fluid. And so, it turns out that it's just not it's good for picking up low-density heat from a large area. It's not great for picking up heat from the high-performance processors. And so, to do that, they're proposing, the immersion community is proposing putting cold plates on the chips themselves and then picking up all the rest of the heat from those small devices with this immersion fluid. And there are pros and cons to that, but that's kind of the trajectory that they're on. Two-phase immersion is coming and going and coming again, maybe. The challenge with that is I need to be able to have a fluid that boils and doesn't get, doesn't allow the combination of vapor and liquid together in a container to get at too high of a pressure. It gets a little complicated here, but that boiling point of that fluid really matters because if the pressure rises even to a third of an atmosphere and I'm in a great big tank, that can create a lot of force on the lid and create a negative very bad situation. So, I have to manage that vapor, I have to manage the expansion and it's a little complicated there and there are other things we actually published me and two colleagues published a paper on this about four years ago, three and a half years ago now that went into some of the details of the limitations of two-phase immersion and it's again going to be kind of stuck at lower powered, still pretty high powered, you know, like a thousand watt processors, but it's kind of stuck there in that region for, you know, thermodynamic reasons. So, really the emphasis in the industry is on the single-phase direct-to-chip cooling because it's ready to scale. We have a supply chain that's very well robust worldwide. We have liquid coolant suppliers that are now able to provide high-quality coolant worldwide. This is what we need because we're at the state where we're producing and consuming hundreds of thousands of cold plates a month and by next year that could climb to maybe close to a million cold plates a month. I mean, this is crazy. You know, in 2022 we might have been talking about 100,000 cold plates a year, right? In the whole industry. So, something on that order, right? And so, we're just scaling super fast and right now the technology that can scale and keep up with it is single-phase direct-to-chip. For data centers, connecting to the grid quickly means two things. Building distributed energy and having a plan for demand flexibility. GridBeyond helps with both. GridBeyond's DERMS and VPP platform unlocks the potential of distributed assets and dispatches flexibility into the right market at the right time. And this is more than software. GridBeyond supports developers every step of the way. From modeling your highest ROI energy mix to orchestrating the full set of assets around a data center. The track record speaks for itself. GridBeyond manages over 1,400 sites and more than 3.7 gigawatts of load across Europe, the UK, and the US. Learn more at gridbeyond.com or click the link in the show notes. If you work in energy or climate, you already know the company's antenna group has helped build. For three decades, they've been the communications and marketing partner behind the industry's breakout companies, category creators, and market leaders. Today, their team is working across the most consequential areas of the energy economy. From data centers and grid modernization, to nuclear, geothermal, storage, critical materials, and next generation technologies. Antenna helps ambitious companies become the names everyone in the market knows, trusts, and wants to work with. They build reputation, elevate executives, drive demand, and turn momentum into market leadership. If your company is ready to lead its market, Antenna is ready to help you get there. Visit antennagroup.com. When you harness the power of millions of DERs in a VPP and put artificial intelligence behind the dispatch, the outcomes exceed conventional generation. Energy Hub's DERMS platform has been trained on years of real dispatch data from over 2.6 million DERs so utilities can plan with confidence. That means consistent load reduction across a four-hour event window, even during the widest, most uncertain peak periods. Utilities target, shape, and verify capacity down to the meter. Unlock flexible capacity that costs 40-60% less to deploy than traditional infrastructure in a fraction of the time it takes you to break ground on a peaker plant. Learn more at energyhub.com. Okay, so now explain the relationship between the cooling architecture and water consumption. What's the relationship between the cooling architecture and water consumption at the data center level? They're actually kind of disconnected. And let me explain that. What we want to do is break down the journey of the heat from the chip to the outside into basically three parts, right? We're going to have to collect the heat from the servers and then move that heat through some fluid, air, liquid, whatever, to a exchange point between what we call the TCS, the technology cooling system, which is the part that collects the heat, to the FWS, the facility water system, which is the part that moves the heat to the outside heat rejection system. And so the heat rejection system is going to be whatever devices that the data center designer has chosen for that climate, for that region, to be chillers, cooling towers, dry coolers, whatever's appropriate for that particular region and power level. And ultimately, that interface between the TCS, the cooling architecture that's collecting the heat, and the FWS is just a heat exchanger, regardless of whether I've got two-phase DLC, single-phase DLC, or immersion. There is typically a plate heat exchanger that interfaces between the two systems, and the FWS doesn't care what was on the other side. What we care about is having the highest possible temperature on the TCS rejecting the heat to the FWS, because I want to move the heat from the processor, which is the hottest point in the system, to the outside. I want to try to move that heat downhill the whole way. Then I can just have thermodynamics carry the heat out without requiring an energy input other than pumps. And that's the most efficient way to move the heat out. The challenge is that in many climates, if the outside temperature rises up above 100 degrees Fahrenheit, 43 degrees Celsius, that's pretty warm. And that's actually warmer than our FWS often wants to be. And so that means we have to push heat uphill, and then I need a chiller. So what we're doing is we're looking at the climate, and we're looking at the heat load, and we're looking at how much time are we spending in the year in a case where I'm having to push heat uphill. And if it's just a relatively small number of hours, I might use something called an adiabatic cooler. It's basically a dry cooler with some pads, and they spray the pads with water. This is oversimplifying. My colleagues in the heat rejection industry will be great at that simplified solution, but basically it's actually an elegant solution that allows me on those days when I need to dump heat out on a hot afternoon, I can do so basically with just a little bit of water and do it very efficiently and get that done. Other climates, I'm going to spend half the year requiring pushing heat uphill, and then I'm just going to need a refrigerated chiller. And that uses a lot of energy. So I kind of went a long ways from your original question, but I want to kind of disconnect. Our goal is to get the best technology on the TCS that generates the highest temperature to feed the FWS, to feed the highest temperature to the heat rejection system so that I can use, hopefully, the least amount of energy to push that heat out into the environment. So that all makes sense, but then what does dictate the water? consumption, the total water consumption of a data center? Sorry, I kind of blew past that, because most data centers don't use much water at all. And they historically haven't used much water. And it's kind of frustrating that this whole discussion is even going on. But here's the deal. When we were air-cooling everything, hyperscalers who are building out thousands of the same rack over and over and over again, their biggest cost is energy input to the data center. And so they were looking at this and saying, well, the most energy-efficient way for us to cool this data center is evaporating water. And they were elegant systems, they are elegant systems, that used kind of like I was describing earlier, basically spraying a material, a filling filler material, and blowing air through it, using evaporation to cool that air down. That's actually a smart idea. It uses a relatively small amount of water for the amount of cooling power generated, or the amount of cooling that's being done. But, for those select data centers, it did consume, relatively speaking, a lot of water, because I'm cooling megawatts, right? And so, for every, I'm going to use a kilogram of water for, every two megawatts or something like that. So, over time, in some communities, they paid attention to this, and to be frank, the hyperscalers were very transparent about this. Google, Meta, Microsoft, very transparent. They were honest about the amount of water they were using. That has kind of come back to now, I don't want to be over-negative, people should be concerned about this, people should be paying attention to these things. I don't want to be negative on that point, but the problem is, the data centers we're building today aren't like that. We're not cooling everything with water like that. Even when we're using a cooling tower, we're frequently pairing the cooling towers with dry coolers. So, we're only using the cooling towers to trim the heat rejection on those hottest afternoons. So, as far as how much water consumption is there, I don't know that number precisely, because every data center is going to be different. Most data centers are built completely closed-loop with no cooling towers even, no evaporation. Some are built with, like I said, some evaporation on the hottest afternoons just for a few hours a year, or maybe tens of hours a year. Others are more dependent on cooling towers, but they're using it more for trim cooling, or just the watts that need to be cooled on those hottest afternoons. And so, what we do know is that, the hotter the water, so using a cold plate and direct liquid cooling at the TCS gives me hotter water to the FWS, which gives me more options on the heat rejection, which allows me to use less water if I have to use water at all. So, the lesson here is do the best I can collecting heat so that I can be as energy efficient rejecting the heat as possible. So, I guess it'd be interesting maybe to separate out then, you know, clearly what has happened over time is that data centers have, more recent data centers are consuming less water on net. You could make the case that they were never consuming that much water, but it's like all a matter of perspective. Either way, on a relative basis, they're consuming less now. And it seems like you've given sort of two reasons for that to be true. One is a different optimization, basically, where like early on, the hyperscalers were optimizing predominantly for energy consumption. If you're optimizing for energy consumption, then you do adiabatic cooling or whatever and do use some water. And now, if you're optimizing a little bit more for low water consumption, then you just choose a different method of cooling and you use less. That's one reason. The other reason being the sort of technical thing that you described of like getting higher temperatures in the TCS so that you have less heat to reject. I don't know if it's possible to separate those two from each other, but like how much of the change in overall water consumption in modern data centers versus previous generation data centers do you think is from this re-optimization versus the technical advantage of a higher temperature TCS? I would say that a large percentage is using direct liquid cooling or more efficient TCS technology. Because if I'm air cooling and I'm not consuming water to do that air cooling in the data center, I'm requiring cold water there to generate the cold air. I've got to have cold water going through the heat exchangers to generate the cold air. And that's going to mean that I'm going to need chillers. And the most efficient way to run the chillers is often, depending again on the climate and many other things, is often with a cooling tower providing cool water to the chillers so the chillers are consuming less energy but I am consuming some water. So it's all connected to this balance, as you noted, between water usage and energy. But if I start by picking up heat with 45 degrees C water at the chip and then my return is 55 degrees C I mean that's 135 or something Fahrenheit, I don't remember the exact conversion. I can reject that heat in most places except the worst day in, you know, middle of India in the summer. I can reject that outside relatively easily, right? And so that has made it without the use of consuming water or, you know, a whole lot of energy even. So that's just made a huge difference. As we moved into this AI deployment and it's one of the reasons why Vincent makes a pretty big deal about this in his keynotes is it's enabled this build out to occur with, you know, optimizing the whole supply chain, whether it's chillers, whether it's dry coolers, or whether it's water consumption. So that said, I do want to come back to the fact that these are really large data centers and some of them have really huge piping infrastructure and there's a certain amount of water required to fill all those pipes and during the installation and startup, we have to flush and clean those pipes and that does use water. Now, it's not all consumed. Initially, I can say we were probably dumping a lot of that water, but now and just, again, remember this is happening over a relatively short time period of a few years, now you'll find that most of the companies are pretty sophisticated about that. They have pumping stations that come out and they are reusing the water, they're refiltering it on-site often, so that there's very little wasted water. But that and so it's still likely that there's water used from the local water supply to do the filling and cleaning but even then, it's not always from the local water supply. It's coming, you know, that water may be trucked in from, you know, Lexington, Kentucky, where one plant is and from, you know, wherever, right? And so, there's just not a really strong correlation to the water being used to fill and operate a data center and actually where it's being drawn from. And so, again, it's a issue with a whole lot of variables to it, and I don't want to say that a community that's concerned about water and has seen their water table drop, that they're not seeing an issue. I don't want to claim that. You know, they must be wrong. On the flip side, I think if you dig deeper, there's probably many, many reasons for that, right? And it's not, it's possibly not because of the data center solely or it's possibly not because of the cooling system. It may be that that water is used to build the roads to the data center, so maybe the data center is still responsible but, you know, it's not a continuing use. All I have to say is this is a complicated question, but on the whole, water use in data centers is pretty low compared to many other sources of consumption, and we are continually innovating, and I believe making more efficient power and cooling systems such that we're consuming less water for the power that we're using. I guess to ask you a final question that maybe steps one degree outside of your area of expertise because it gets more into this, like, question of community acceptance. You know, I think this conversation has, um, has clarified why this is challenging, because on one hand, you can look at the raw numbers of, like, how much water have data centers consumed historically and then currently, and again those numbers are changing, like, it's falling, the numbers are falling, but you can look at the raw numbers, but, you know, this concern that has arisen and the fact that water consumption, perception of water consumption is one of the two or three big issues right now in the sort of data center backlash. I think it's difficult to combat given the nuance associated with it. It would be easy If it weren't nuanced, and if every new data center could just say, I promise zero net water consumption, right? But in the absence of that, and here's how I'm going to do it. But in the absence of that, it's complicated. And it's like one of these things that political messaging doesn't do a good job of describing. So I wonder if you have a view on either from a technical and architectural perspective, like should modern hyperscale data centers all do X, Y, or Z? Or from a messaging perspective, what is the right way to communicate the relationship between data centers and water consumption moving forward? A tough question for sure. But in the past year, I would say, you know, Microsoft has been a great example of messaging. They got out front of this question. They were very aggressive in messaging to the community in Wisconsin, where they were building a large data center. I felt that they were messaging. They were messaging very clearly how they were going to cool and how much water they intended to use and how they were, you know, achieving this. So kind of doing what you suggested there, unfortunately, I still see a backlash to that data center. And I still see water being brought up as a point. And my feeling is that that, number one, I think it's table stakes to take the approach that. Microsoft did in that case. And I think as an industry, I think we're getting better about that. Maybe a little too slowly, but I think we're getting better about being really transparent in the communications. I think we could do better literally just inviting community in to see how this works, if that's possible. There's a whole lot of safety and other things and some privacy concerns. But, you know, I think that would help people see, OK, I get this. On the other hand, I think being honest about, hey, building a great big building that's going to consume 500 megawatts of power just uses water. It does. Whether it's just keeping the dust on the construction site down or preparing the roads or, you know, pouring the concrete, whatever, right? It's going to consume water. And so there's going to be a period. Where water is consumed, building a huge data center campus and communicating that might help. But overall, this is an opinion or a belief. Again, this isn't my area of expertise, but, you know, I'm in the waters like you are. And seeing that, I wonder if a lot of this just isn't a general feeling of discomfort of this new technology that's being built out all around. I think we're all uncertain about how it's going to play out. And it's pretty easy to take it out on these big nondescript buildings being built and consuming resources, right? And so I think what we can do as an industry is continually perform outreach, try to explain what we're doing, be as transparent as possible, try to get away from all the NDAs. I understand sometimes why they're there. That just breeds distrust. And we all know that. Again, I understand the reason for that sometimes. But just being transparent. And then there's not much we can do about communication about AI and its application these days. That depends on a few loud voices that you and I don't have any control over. But, you know, but really trying to work as an industry to be transparent and hopefully bring focus on the applications that people do like. I think, you know, autonomous vehicles and autonomous taxis are being used in a huge number of communities. And a lot of people really like them. You know, Waymo and cyber taxis now and other options. Those wouldn't be possible without these data centers. And, of course, all the functions on our phones and whatnot. So don't need to go down that path. But so it's I think this ends up being more of a human issue than a technical issue. Even then, a communications issue. And it's going to take a little while to resolve. But transparency, I think, is the first step. All right, Tim. This was really interesting. Very much appreciate your time. Thank you. Tim Shedd is the founder of Abulian Insights. This show is a production of Latitude Media. You can head over to LatitudeMedia.com for links to today's topics. This episode is produced by Max Savage-Levinson. Mixing and theme song by Sean Marquand. Anne Bailey edits the video version of the show. Stephen Lacey is our host. He's our executive editor. All of our episodes are on YouTube. Subscribe to Latitude Media for episodes of this show and Open Circuit. You can also find the audio version of the show and not have to look at my face anywhere you get your audio podcasts. I'm Shail Khan. This is Catalyst.