Go back

EP 180: HPE Reflections on AI Compute Racks, Analog Analog Analog Semis.

48m 23s

EP 180: HPE Reflections on AI Compute Racks, Analog Analog Analog Semis.

In this episode of The Circuit, Ben Behren and Jay Goldberg discuss insights from HPE Discover, focusing on the engineering differentiation in AI infrastructure. Behren highlights HPE’s hybrid cooling pod, which mixes air and liquid cooling to fill a customer gap, and ultra-thin liquid-cooled trays that pack more Vera Rubin GPUs per rack, maximizing compute per unit space. This contrasts with the PC era where OEMs had little room to innovate. The hosts note that Nvidia’s inability to impose a uniform reference design has allowed ODMs and OEMs to engineer unique solutions, particularly in cooling and power management, benefiting NeoClouds and enterprises. They debate the role of NeoClouds as a middle market between hyperscalers and long-tail enterprises, with Dell and HPE balancing semi-custom designs to meet demand while managing margins. The conversation also explores the growing trend of on-prem AI factories, where enterprises buy a handful of racks for sovereign or latency-sensitive workloads, potentially revitalizing the on-prem server market. Overall, the hosts see this as an engineer’s dream era, with intense focus on efficiency and differentiation driving value for customers and vendors alike.

Transcription

7873 Words, 43317 Characters

English
[MUSIC] >> Hello, everyone. Welcome to another episode of The Circuit. I am Ben Behren. >> Greetings, programs. I'm Jay Goldberg. >> So, there wasn't a huge news week. However, I was at HPE Discover and had a number of interesting observations and takeaways. I sort of joked if anybody saw on Twitter that this is my life now and I see an open compute tray. I put my reading glasses on and my phone into magnification mode and I go look at silicon, names of vendors on silicon, MLCC's, power capacitors. You name it, which is partly the, I like to see how these things are designed and built and the engineers on the floor are generally very excited to answer very technical questions. Something that I greatly appreciate about these shows because they're not just, oh, this is what it can do. It's like, oh, that's why this is where the power thermal is. We designed it this way so that we could squeeze out x, y, z, even efficiency or network latency. I'm like, this is the greatest conversation ever. But there was a couple things I'm taking away from this and I don't know how to sort of frame this in the analysis because it is technical. But basically, there's a discussion out there around how everybody designs their infrastructure. We know Amazon with AWS or Microsoft with Azure or Google GCP, they design a lot of these racks. Just by, you know, Nvidia GPUs and go with everything or even for their TPUs, right? They are engineering the crap out of these racks for something. You know, again, everybody's variable changes. Maybe it's performance, maybe it's latency, maybe it's efficiency, you know, you name it. Maybe it's to use less water, maybe it's to use a little bit of water and more fans. Like everybody takes an approach. And I think that's interesting. And again, kind of going back to our Computex take, you know, I did this same thing. It's just harder when you're looking at a laptop, right? Even in Computex in those days, like very rarely where they're like, let you just see the inner workings of a laptop. But even if they did, there really wasn't like that much differentiation, right? There wasn't a Acer or Dell or Lenovo, like is just eking out drastically better performance with their Intel skews or drastically better battery life, right? It was roughly the same. This is very different. Like people are really architecting solutions. And so for example, like HP has this hybrid design, they call it. It's a four rack pod and it is a mix of air and liquid cooling. So what it can do is it obviously can do more cool more watts than an air and an air only implementation, but it can't cool as much as a pure liquid. But that's actually very interesting because, you know, it still has a rear heat exchange. It has, it's built for better airflow airflow. It still has per rack, a very small little closed loop liquid system. And I just thought that was interesting because, you know, their point was, you know, not everybody wants to just go pure liquid. Even in some of these implementations, there is an opportunity to have variation in that. Again, if power is your key limiter and space, for example, right? Because the liquid cooling systems are gigantic. It was just an interesting approach, right? And I hadn't seen kind of this, you know, we've designed a hybrid implementation because it actually fills a little gap that a number of our customers, you know, sort of want. And that took very specific engineering across the board with that kind of implementation in mind. So again, we've talked about this before. I'm intrigued by this moment is an engineer's dream. There are so many problems. These guys are so excited to be just working on what they think are just fascinating problems. And I agree they are. But my point is it's just, it's super interesting as an analyst to be like, okay, Dell or HP, explain to me what you designed. That's different. Like what engineering little marvels or nuggets or innovations did you do? Is different than a competitor because there's far more in this opportunity for an OEM or an OEM or a hyperscaler, even like we talked about with Nebius, why is Nebius designed or sucks? Because we can make a more efficient. We think we can do a better job than other people because our engineers are focusing on a spec. And I just think that's an interesting cycle we're in. You know, whereas you would have said in the PC era, everybody's kind of the same. There's little differentiation. You just ship ship silicon and that's it. That's not quite the case with AI infrastructure. And so that was one of my takes I thought was interesting. I, okay, I can do this. I'm going to muster up the energy to get excited about racks. It is a weird, a weird time we're living in where suddenly people are interested in Dell and HP to such a degree. I mean, their stocks are double or triple this year. I think if you just think back a year ago, there were a lot, there were a lot more questions about to what degree the OEMs like them or more importantly the ODMs in Taiwan could actually differentiate. I think there's a lot of fear last year that Nvidia was just going to do a reference design. Everyone's going to follow the reference design and the scope for that differentiation wasn't there. And that changed. I heard that, I predicted that, I was wrong. Because we've gone the other direction where we are seeing the ODMs innovating, engineering, heavy, heavy engineering coming up with new solutions, especially around cooling and things like that because it's become so important. And I think it's also worth noting in that it sort of says something about Nvidia's ability to impose a reference design on everybody that didn't work. And so I think for some customers, taking Nvidia reference design is going to be enough. And for others, there's a lot more others who aren't interested in it and want to do it their own way. I also wonder to what degree it shows a maturity on Nvidia's partnering program. I think they realized that they were up against something. I've heard this too is that they have sort of overhauled how they deal with their ecosystem, their ODM partners significantly. Something changed inside of Nvidia too where they're like, let's foster more differentiation and give them some room here. So they did something right there. I think there's that behind the scenes as well. So good for everybody. I'm glad, you know, it's a good time. There's a lot of interesting stuff, even if it's like low level stuff that a few years ago nobody cared about like cooling, like analog. Right. But the fact that there is differentiation out there is a good sign. Yeah. Even like, you know, I had seen, you know, I'd say Dell does some of this too, but the standard rack I saw from Dell and a liquid cool solution looked very similar, a little more similar to Nvidia's reference design. But HPE has these solutions. They call them Kray. These are the Kray design trays. They've been doing this for a while. It goes back, I don't know, first server was a cylinder that looked like the intro-silinger of Iron Man's chest. And now they've evolved. But what was interesting was they were walking me through, it's like one, I took a picture of this thing. It's this gigantic kind of liquid cooling thing that looks like an engine of a monster truck. It's sitting on this left side, right? But what they did was they slimmed the tray that goes into that. That's liquid cooled that has all the Vera Ruben specs on it, right? And many of capacitors per which you counted in the picture I sent you for power management, thermo-management. But it's so thin, they're like, we can pack this many more Vera Ruben rack, Vera Ruben's you know, GPU CPUs into our solution because it's so thin and we've optimized for that. And therefore this system, while it can do 1.61.8 kilowatts per, they can pack a whole lot more of them. So if you're again space limited and they're designing this thing that gives you much more compute liquid cooled per unit of space, like that was their design, you know, challenge. And it's just, I guess I just, it's just clever to see how folks are going about this to again maximize compute tokens per watt efficiency in a design liquid cool for how many kilowatts per rack via the system because they figured out how to make it so thin. And that's just where I was like, this is just, this is super interesting. How again, how everybody's kind of solving these problems. And then, and they had another one that our friends that served the home Patrick Kennedy wrote, which I thought was cool, where it was an eight epic CPU. So again, same design, so same thinness in the tray. These could go into, you know, it by themselves as a CPU rack, but basically it was a liquid cooled CPU rack with a massive amount of memory stacked on top of DIMS, all coded in copper to cool the entire thing, still in this insanely thin profile. Like it was just, like the engineering of it, again, any engineer would look at that and be like, "Tell me all about this. This is incredible." Like how did you do this? Like just fascinating stuff, right? Going on to assemble these racks in creative and differentiated ways. >> Yeah. I mean, those are some pretty incredible photos. >> I keep coming back to the business side of this. It's really interesting that Dell and HP are having this moment here. Because if you look at the share prices of, like the Taiwanese, the Taiwanese ODMs who are sort of competing with them, sort of not, their share prices have been flat, except they all spiked in the last, since basically right ahead of a contour right at Computex, because there's so many bullish signals that Computex for that ecosystem. >> Right. >> And so I wonder to what degree they can, I'm just curious how this plays out between those two groups. Because the ODMs tend to be less customer, or like they work with the hyperscalers more directly. And so they're benefiting from that, but they've had that problem of differentiation. Dell and HP I think are much more geared towards the broader enterprise to, and the Neal Clouds and maybe some of the hyperscalers, but still there's a difference in their business base. So it certainly is some sign that we are seeing more companies, enterprises take direct ownership of their AI stack, which is something you've been talking about. >> Well, and so okay, but here's another reason where I think this kind of gets interesting. So bear in mind, right? You are right, the vast majority of ODMs, sorry, hyperscalers, use ODMs like Whistron and Y-Win and you know, Quanta and a handful of others. Now while they can scale and compile for references starting to do more servers in there, we're particularly a notebook OEM, but the Taiwanese ODM ecosystem does this. Now there's two things. One, you get a little bit over the sense that the hyperscalers are capacity constrained there, right? That those ODMs are scaling up as fast as they can, but they're still enough demand that perhaps HPE and Dell fit into that. But two, if they, let's, and again, this is why I think the design of the rack is super interesting. If they give the NeoClouds a tokenomics advantage, a better revenue per megawatt, that then sets the NeoClouds up for both themselves and also to be their hyperscaler customers, right? So that's why I think it's just interesting, win again, your metric is like, I gotta squeeze out all this juice so that I can make the most money of compute per megawatt, I just wanna pack as much in, 'cause that impacts my dollars, it becomes an interesting competitive infrastructure conversation, right? It's not just by all whoever, it's I'm gonna make more money as a NeoCloud or an infrastructure landlord, right? What not, if I buy from, in this case, let's say HPE or Dell, because they're optimized, right? For better revenue per megawatt based on the efficiency. So I just, again, right? And then again, that matters for their economics for either their customers or the hyperscaler that might use them in a bare metal environment, becomes a detractive lead, if you will, if their economics are better. - Yeah, and this is a debate I hear across this corner of the ecosystem, people who are building systems for who are building server systems in general. And how do you sort of prepare, where do you invest? Like what should be your focus? It's clear that the hyperscalers, they've had an established method of dealing with the ODMs for several years now, but what about everybody else? And the NeoClouds are the big sort of wild card in this industry. Like I've had a lot of these conversations where if you're Dell or somebody, someone else smaller than that, lots of players in the ecosystem, you've had your relationship with the hyperscalers, it's very hard to get in there. They wanna do everything themselves. They wanna beat down margins as much as possible. So that's kind of, you either have that business or you don't, the long tail enterprise is big and complicated. You really need to work, go through someone like a Dell or HP to participate in that. But then you have the NeoClouds in the middle who are a significant source of volume. And they are unlike the hyperscalers, they just can't do everything themselves, right? They can't design, they can't necessarily design the servers themselves, they can't design some of the more complex subsystems. They're gonna wanna work with more partners. So that means it's good business for those people. Like Dell, it's good business. It's higher margin, you're providing more service. But then every question I get asked about them is like, well, how much should I invest in these NeoClouds? How long should I invest in them for? Are they gonna be here in five years? And you know, even for some of Dell's size, I think it's an important conversation because one of Dell's priorities since, you know, Michael Dell's building systems in his dorm room was how to sort of design things that are semi-custom, right? Every customer wants something that's like theirs, designed perfectly to their specs. But if you're the one doing the building, doing something 100% custom, it can be very expensive. You lose economies of scale. And Dell is very adept, I think, it's sort of balancing that. Making a system that is composable to some degree, so you can get some customization, but not pure customization. And for the hyperschools, that doesn't work, great, 'cause they're like, we're gonna specify everything down to the color of the PCBs. Other customers can work with that. But then if you're Dell, you, like, I've done stuff. I've talked a lot of people at Dell about this is they're very, very concerned about not letting the catalog get too broad, right? I forget the number of, they have a very large number of skews, of server boards, designs. And it's like dozens. And they don't want to add another one to that, because they have to optimize for manufacturing and they're constantly thinking through these things. And for them, like, the NeoClouds are a tough problem there, 'cause there is some degree of customization, they want more skews, they want all these interesting things that are on the show floor, how much your HPE or Dell really wanting to add to their product portfolio, how many skews are they gonna add? And what was interesting on Dell's last call is their margins were up, right? And so we have to assume that they've been doing more customization, they've been broadening the offering, but they're able to do that and keep hold on to margins. I think that's been, I mean, to some degree, it's just a function of like, there's just immense demand, and so you can sell whatever you can, that's helped. But I do wonder like how they've managed to balance all the competing needs of customers, to semi-custom and how much they seem to be navigating it pretty well, but. - Right, that's where I kind of focused on. - Yeah, no, and you're exactly right. But so I wanna talk about kind of this other class of customer, which we've kind of talked about before, I mentioned this after I was at Dell Tech World, and now after talking with HPE and all sorts of customers on the floor of Big Enterprise, and HPE is hearing this too, as it's all, there is, there is, well, I was gonna say, I'm more convinced and convicted of my thesis that some AI infrastructure moves back on-prem. So both these companies have divisions of their kind of sales and strategy team that's called private AI factories. So it's essentially some workloads that need to remain sovereign, again, not all, but some are worth keeping local tied to your storage at the edge so you're not dealing with latency of moving, massive amounts of straight storage to the cloud, yada, yada, yada. I don't know how many, but here's what gets interesting. If you, I was talking to some of these guys about like, all right, what's the size of your data, like your on-prem data? And we're talking about petabytes. Now if you just do the math, and again, that they know that's an 100% data's growing, 'cause it's not static, so if it's X petabytes today, it's going up, but regardless, if you just do the math on, let's just take a Vera Rubin Rack, for example, and they've got X number of petabytes that they wanna keep local and run agentic on top of, it's not that many racks. Like you're talking less than 10 for sure. It might be less than five, to be honest with you. That becomes a fairly reasonable, and again, I know we're not talking 100, like, you know, the same as a data center scale, but the long tail of the enterprise, right? Somebody buying ARAC to five racks, right? Or whatever, can still power that. You know, your power bill's gonna go up, but you've still got that. You're not building a little data center pod, right? In your parking lot. But it seems reasonable that they can get a handful of these racks are smaller, and really keep build their own small AI private factories. And it feels like it's gonna move in that direction, which so it's again comes back to all these conversations we have about HPE and Dell, right and whatnot. And even in video, is the like, your customer is, the clouds. And if this is true, right, then their customer also becomes the long tail of the enterprise. And while that's different dynamics, it's still a big market. There's lots of them. They might need again a handful of servers. They're probably still going to refresh that every five or six years, right? In some degree, that's just added dollars to the AI server tab in billions, possibly hundreds of billions of dollars globally. Well, ex-China, they might not be able to sell in China, but you know what I mean. So I think this is interesting as another market. Now, certainly, there's a whole lot of enterprise software conversations that come into this that CIOs and CTOs are thinking about. But the point I want to make is that on-prem feels like it's going to be a thing. It does seem like both HPE and Dell are gearing up to that to go re-ignite what was a on-prem server business that they had for many years with those customers for what we just loosely will call these kind of little on-prem AI factories. Yeah. I mean, I think this is going to be a big subject. It's not a predictive. We've been talking about this for months. But this degree to which AI moves on-prem or to the edge, that's what we seem to be calling it now. I think I'm a little more, uh, wait and see about it. Just because I just don't think we understand the workloads well enough. Really, I think there's everything's changing so quickly and so constantly that is hard to say. I still think there's a big, big room for the data center. This is going to be AI's going to remain complicated and it's going to be all over the place for a long time. And the original thesis behind moving workloads to the cloud, cloud computing 1.0 from 2010 still holds. There's still a lot of reasons why you want to do things on-prem. Now, do, do, does AI change that? Certainly around privacy and data sovereignty. There's, some issues there, but we had those before. We had those before. We had those, right. That was, that was a big debate with AWS 15 years ago. Right. And they got through that. Um, the, the models themselves are changing so quickly. We obviously have anthropic in the news lately. A lot can we use fable mythos? Right. There's an argument there that says that means everything's going on-prem. I, I think we're, I think there's, I think everything's moving so quickly. It's too hard to make solid predictions here, other than we're gonna use a lot more compute. And like somebody sent me a very bearish a-i-take. I seem to get people sending these to me all the time now 'cause I guess I'm the AI bear. Right? And like the thrust of this argument was, it's all going on-prem. I'm like, that's not true either. Right? - That's not true. - This article in particular was, I'm not even gonna mention who wrote it 'cause I don't wanna feed the trolls, but like the whole point of it was like, the data center that meta is building in Louisiana is gonna be a colossal failure 'cause we don't need centralized compute anymore. I'm like, no, we need centralized compute. Like, set aside legitimate questions about what meta strategy and AI is. Like I'm pretty sure if they build this massive data center, the size of Central Park, somebody is gonna find a really good way to use that to advance the front here. So, I'm coming down on, it's like all of the above. There's gonna be tons of stuff on-prem. I think we're gonna have to monitor exactly. I don't think we know which way it, where it's gonna end up. There's lots of arguments either way. And let's not forget eventually, eventually we're gonna get a lot of this on PC's and phones too. - Yes. - Probably a couple cycles away, but it's coming. - So, what I think about is interesting too is, when you listen to Dell and HPE talk and both of these will make sense in the context of kind of where they're both invested as what we'll just say is like the tip of the spear, right? The thing that ignites the conversation that then leads to a whole slew of other infrastructure questions. So Dell leads with storage. And I think there's a very good reason for this because to be honest with you, to your point, what the argument for enterprises were in, workloads to cloud and whatnot, two bonds with you like, yes, it was security, but it wasn't security to the extent that I think people, like, fully understood, right? The hyperscalers have a very good way to keep you isolated, right? And firewalled from other customers. Like, that's kind of less the issue. Where I think this gets interesting is, there is a very good reason to not constantly be pushing all of your data up to the cloud, right? There's a reason why storage is created and lives at the edge. And in this kind of agentic future, all of that data that's created at the edge needs to be labeled and structured in real time, which is where I think the edge PC will play a role. It's basically like we talked about last week. Apple is still indexing my life, by the way. It's been two weeks, Syria is still indexing. Something on your computer will constantly be indexing and organizing that data as you create it. And then it's gonna go somewhere. It might live somewhere on yours and then it's probably gonna go maybe to a central knowledge base for the enterprise. But that doesn't make a lot of sense to keep moving to the cloud. So just on the basis of a knowledge base or information of data that's permissed, observed, organized, structure protected at the edge, that makes sense to stay at the edge. So then you're gonna ask yourself, okay, how do I let agents run across this? Well, that's where again, the kind of hybrid cloud plays a role that I think is interesting. So without going fully and I put a whole report about this on the agentic storage shock to the industry, there's that. So Dell leads with that and then it leads to, let's have infrastructure conversation. HPE on the other hand, and this is brand news to me this week. And to some degree, I don't know when HPE decided to take this pivot, but they're really leaning into Juniper and I had a handful of executives essentially tell me, you should actually think about us more like a networking infrastructure company now and that we're gonna probably face up against Cisco in some situations more so than we will necessarily against Dell. Now that's somewhat true. You're still up against Dell, but the reality is they're leading with you're gonna start to make network questions about running agentic in your enterprise, right? You're gonna need more, you're gonna have to redo the networking. So taking 1,000 foot view, if you just take both things that I just said, the enterprise itself, just by nature of wanting to do anything in agentic like we're talking about, does have to do some refreshing of its infrastructure, whether that's again, the PCs that do these things like you said, what happened on the edge, the network in the middle, the place that it stored, whatever. I firmly believe that that's gonna happen, right? Regardless of where all these workloads go, infrastructure needs to be changed. But I thought it was interesting that you take those two things and HPE wants to lead with, you're gonna ask networking questions and then we're gonna bring infrastructure along for that ride. How can we solve your compute? Dell's gonna be like, you're gonna ask storage questions. So we're gonna go storage and then we're gonna talk about all these other things that you need. And they're happy to use, right? In video, then VLink and ride that tide where HPE is leaning into Juniper. So two different organizational arc strategies, tips of sphere, both solving different problems, both aligned at this idea that enterprises will need to upgrade infrastructure to do these things that we're talking about. I just found that interesting, but both of it is centered on again, this just idea that more agentic will happen somewhere in those four walls and then how much it goes to cloud or on device. And again, that's an orchestration software layer. Like that's not even solved yet, so what not. But anyway, I was just kind of my guts of it, like the interplumming of organizations and how both these two companies are coming at it that I thought were interesting. And both have a range of implications to think through when you just again think about data center or enterprise infrastructure, IT infrastructure, I guess. - I just wanna say for the record, I win. - On networking. So you're gonna take the networking set. - So, go ahead. Go ahead. - Have you all excited to talk about networking? In a minute we're gonna talk about analog. Yeah, I win. - Jay does win. I remember when I said the networking. Anyway, all right, anyway, yes. - Networking, so I agree. Networking is gonna be really important. I am, how do I put this diplomatically? I think Juniper is in Proofit camp. Like, let's see if HB can really do that with that. But sure, networking is gonna be a big deal for all this. And I think this is one of the barriers at the old days. OEMs are up against trying to sell the hyperscalers, as the hyperscalers have a very, very specific way that they want to design their networks. This is a big part of their reluctance to get more reliant on Nvidia, 'cause Nvidia wants to push NVLink. The Amazon, Google, in particular, Microsoft, to a lesser degree, really, really care about networking. It matters a lot to their business. And so if you're trying to sell into that, it's much, much harder. That's one of the things that makes harder. Now, networking is important, 'cause it has a real impact on inference costs, training costs certainly, but inference costs too. So it makes sense that the Neo Clouds need something here and so do long tail enterprise. They probably can't afford sort of the really custom way that's like Google does it, but they don't necessarily need it, but they need better networking. - Yep, nope, agreed. Anyway, I just thought that was kind of an interesting tip of the spear like how both of them, and I agree with you, Juniper is good. I mean, I spent, I don't know, a good 25 minutes with the guys from the Juniper team who made, I tweeted this picture, I sent it, you know, to you two, the 1.6 terabyte switch that they made on back at Tomahawk 6. And I was like, what, you know, this part of this is broadcast, some of this is you guys, like just explain what Juniper's bringing to the table. And you know, it's just kind of interesting for them to talk about 'cause it really did show where they need the networking, ASIC and your Silicon guys where they don't to pull these sort of things off and then really how they're playing optical 'cause there's a lot of Juniper designs some things versus just controls the full stack of Silicon like to some degree, Cisco does, for example. But anyway, that's an evolving storage. I thought it was, I thought it was interesting. But those two things I do are I think are creating these interesting conversations within enterprises about, you know, what is our IT infrastructure gonna look like at a compute, at a plumbing way? And that's not resolved. I just, I feel this conversation is bubbling up as both Dell and HP are telling me what they're hearing from customers. This is bubbling up into something. - Yeah, there's something here. I was listening to the AdLots podcast and they had Jeremy Granthamon, whose famous fund manager noted Contrarian and someone who is much more bullish about, much more bearish about AI than I am. And he made a good point though. He said if you look at the big companies who are leading the AI race, the hyperscalers, Google, Amazon, Microsoft in particular, they all had their own sort of spheres of influence. They all had their own sort of quasi monopolies for years. And now they have sort of thrown that out and they're all competing for the same thing, right? They're all in this big arms race for AI, stepping on each other's toes in a ways that they really didn't before. Like, yeah, AWS and GCP competed, but like, or Azure was there too, but like each of those companies had very strong other businesses. And so I can understand if I'm an enterprise in this environment, I'm gonna be a lot more reluctant to give my data over into that because I don't know how that all is gonna shake out. And I really, really, really don't wanna, like AWS for years was very good at siloing off data, like you just said. I, that may change, right? And so if I'm an enterprise, I have to take that more seriously. I really don't want any of these people training their models on my data. And there's, you know, a non-zero risk that that happens. And I think it certainly favors, favors more on-prem and all of the complexity that that in-- - I agree. - I agree. All right, so to wrap this particular bit up before we talk about analog, I think what I've landed on now, having kind of all these fresh conversations around how enterprises are gonna go do this is it's going to be a multi-cloud. So it's not just gonna be one cloud primarily and it's gonna be a hybrid cloud. And you have to think through kind of that beneficiary map differently when you land on that conclusion than just all these things are gonna run in one hyperscaler or even in multi-cloud, one to two hyperscalers. You now have to think about that software stack all the way down into a hybrid cloud and local cloud environment. And I think that's largely directionally gonna play itself out. But it changes the landscape some. So I think that's interesting. - Just for the record, I agree with what you're saying, but I'm much more in the TBD camp. Like I think a lot of those things may-- Anything's on the table now and the final shape of it is still to be determined. - Yeah, that's fair. - Okay, analog. So, okay, an analog in two ways. So, okay, I'm doing. How is it possible? I'm gonna let you explain this to me since you live analog. And I'm admitting this in all for fairness, that analog is supremely interesting again or possibly for the first time ever. Like I can't ever, when I was at Cypress in the 90s, I remember talking to the dudes parallel to us in analog and I was like, it's some coils, man. And sure, it's in the set, like I'm sorry. I think logic is fascinating. - Like just so everyone's clear, the person who said coils has been. So when you come out as a Twitter, that was Ben. Pickin' a fight with some very dangerous people. - Okay, fine. Now listen, I will also say I'm not like unentrieg by these things 'cause I did actually attempt to assemble and build my own electric guitar pedal, which is a bunch of capacitors. Okay, so fine. Again, I just like, okay. All of that to say, it's actually super interesting when the problem has gotten so big and I use big in absolutely every way possible, right? The substrate is thicker, the chips are bigger. The power that's going through your board on that PCB is more intense. And so the way that you have to put, organize your power layout is actually super interesting because of the problem. Like it's a different problem, I guess is what I find. Before analog solved a problem, but it was like, I gotta manage my wads, it's a laptop, it's a smartphone, like fine. We're talking about a whole different things and then it gets even more interesting when you start looking at 800VDC, which I think is causing probably one of the largest content attachment growth stories for analog. I mean, really than anything. I mean, I know automotive has a good chunk of these. We don't get to stare at the guts of an EV, but this one gets super interesting. Like there is a lot of analog content pull, just stacking of capacitors or MLCCs are getting bigger and thicker and whatnot. So anyway, analogs having a moment, it's super interesting when you again come back to how they're designing these things for power management, how much little bits of logic and p-mix and microcontroller processes are on there, but really how much stink and analog goes into a power management board for, anyway. It just bends, walk away of more appreciation of a very complex problem and the way the analog is helping solve it. - And I think it's, we also have to be clear, it's not just analog, it's also passive components, right, which is the even weirder to me, 'cause like analogs a whole, like at least those are semiconductors. A lot of what we're seeing also now is passive components, like MLCCs, like capacitors and resistors. And it's just like, it's, and that, I mean, those are, have been very sleepy, very overlooked, very perceived as commodities forever. - Well, and to your point, magnets, the, you know, gan and sick parts of this that go in, right, to help like largely those are silic, like those are periodic table components, right? It's like, they're getting bigger, they're getting more, like it's anyway. - Yes, that's good. - Yeah, there's a lot happening in analog and passives right now. The passive, passive component story is just, is pretty straightforward, it's just like suddenly we need a lot more of these. The analog is we need more of them, but we also need different categories of them and we need better, higher spec parts. So there's a lot of fun stuff happening in that space. I think that the driver is, it's more than just the 800 volt, the transition to 800 volt data center. It's the reason that we need the 800 volt data center is the next generation of racks starting with Feynman, really, and the Kiber rack that goes around it. - Yeah. - Is, I mean, those things use so much power, right? you know, just like incredible amounts of power. the kinds of power that people would never even contemplate five years ago. And getting that to work is very challenging technically. And technically in ways that people, we haven't really thought of for a long time. But the industry really hasn't thought of an a long time, it's sort of taken for granted, because so much was standardized and like there was pretty well laid out path or doing it. And now the numbers have gotten so big that it's sort of forcing these multiple layers of transition. And now we have to care about these things, because it's a pretty significant overall driven by these big, big power hungry systems. Yep. Which is also increasing, you know, PMIX, FPGA's and some of these boards. I have some of this conversation with TI around 800 VDC. And it was interesting because like they even have these solutions that you have to think through, right, in that much power or thermals you did in 402. But the way which you might need one of these trays to be swappable so that it can actually manage its current so that as somebody goes to turn off a switch and it's not going to shock and kill you when you have to go and operate, right, inside. So just again, like a whole slew of new problems. And being solved in very interesting ways that require a lot of, like you said, like passives, different types of materials, coils, big oil coils, I saw on that network switch board, the chancers of a little rack and coil, wrapped in coil. Like these things were just, anyway, analogs have in a right moment, you know, all the, and I love that there's like, and I see some interact some of these guys on Twitter, like analog engineers who are like, people just want to ask me questions about analog and I'm so excited to talk about it because no one has ever wanted to talk about analog before. And it's interesting again. So analogs have in a moment. I really, I cannot think of a time in which analog has been this exciting. It's back, probably back to the original days of semiconductors when all everything was passing. That's right. Analog. Hence the reason why my peers at Cypress were so excited and I didn't, I didn't understand why. And it's funny too because like, it's analog is hard. Like that's the thing. It's hard. It is. It is. It is. It is. It is. It is. It is. It is. It is. Like you, you, you fail out your analog design class and then you become a digital engineer. Like, so you think you think the analog guys, so the analog, the analog guys are the black coffee drinkers of the coffee. You judge everybody else. You're like, you're just not cool enough, buddy. Right. I, it's hard. Like I've always, like, I come with this largely from the RF side of things. That's sort of my background. Like, and to me, like, talking to the RF analog engineers, it's really hard to like, it's hard not to think that what they're doing is just some form of black magic. Like you have to have such a high degree of faith that you're like, oh, I'm going to do this signal here and it's just going to go out in the air and something's going to happen and then it's going to get received over there. We don't really quite understand exactly everything because there's too many variables. It's just going to work and you're like, come on. That's like, that's the ultimate faith and a higher power that all this is going to work and you know, and it really is some of its, it's, I mean, a lot of it pushes, pushes the laws of physics. It's pretty, it's pretty fun stuff. Again, you're 100% right. And in fact, I do want to make this point because it was interesting. So I did this piece. Man, was it last week? I can't remember on, on the role that Agentic plays in EDA and one of the points that the folks at Kate and so I was talking to and I think this is interesting was, you know, because of everything we described, like maybe people decide to start doing engineering again and become analog designers, but for the art part, that demographic is aging out, right. There's not a lot of new people coming in to do, to do analog. And so the role that an agent can play in domain expertise to help right offset some of that in Agentic EDA was kind of where they were like, this is a problem emerging and we see the value of, you know, kind of these, this domain expertise of analog to exist in kind of these agents that they'll charge more for. If again, that plays itself out and younger, kind of a new fresh breed of analog designers comes hard to find, right. Agentic can kind of offset some of that in the EDA process, which I thought was interesting because it's true. Like it's, it is hard. Like most of the kids I know, really even globally, right, who are getting into, these are going into logic, not, not necessarily analog. Yeah, it's hard. Like it is. You take those analog classes in college and you're like, what I'm having. You start reconsidering your life choices, right. So it is, you know, but that's why I'm so excited to see this moment is because there is lots of interesting stuff happening here. And from companies we don't, we probably have never or very, very rarely talked about on this show. And there are suddenly, you know, front and center and everybody's minds because they're vital and crucial to what's happening next. Yeah. Anyway, I love it. I am, I'm working on, I tweeted this the other day if anybody saw it where I kind of, I hadn't seen it charted the historical semiconductor rent annual revenue with the, the now drastic increase in the next three years of, of, of growth where basically kind of looks like a spike straight up from what was a very slow and steady decline. And I titled it, you know, dear Jensen, thanks, this, with love, the semiconductor industry. But, but really what makes this interesting, this is kind of the point I want to make in a series of things I want to, I want to lead into, in some of our researches, just just the GPU is responsible for this. Like, this is the thing that's created this tsunami. Every other category is in a massive lift because of it. Analog included. And I just think that that kind of pull of, of what I'm just calling the GPU tsunami, lifting everybody's boats, magnitudes, like not even a little bit like we're talking multiples. I mean, I saw an estimate for power semis could be 20% increase of, of, of increase of, of content per board. Like, all because of this kind of AI moment fueled by the GPU. So anyway, I'm sorry, 20 times. Did I say 20? Yeah, I was going to say yes, it's not 20. 20 times. 20 times. Which is just, I mean, think about that. Like just, just like the amount of demand. It's unbelievable. You post the photo, you sent it to me too, is like a Vera Rubenboard just like in the small section you showed me there were 45 capacitors per GPU. Like, yeah, it's just, and that's just like, and the rest of the board, there's a lot more to it. So there's, there's a big, there's a big uplift here. Yeah. Yeah. So if like, so maybe every bit of semi-connector content is just being, you know, multiple times more because of this AI moment. And I just think that's, that pull is just fascinating. So what a time to be in semis? Yes, as we will know and are excited about. So I will, I will chain my interbearer. Yes, it's a good time to be in semis. Okay. All right. We'll have your bear thesis another another day. All right, thanks for listening everybody. That's our show. We will talk to you next time. Thank you, everybody. Tell your friends, click like and subscribe.

Podcast Summary

Key Points:

  1. HPE Discover showcased significant engineering differentiation in AI infrastructure, moving away from uniform OEM designs; HPE’s hybrid cooling pod and ultra-thin liquid-cooled trays for Vera Rubin GPUs are key examples.
  2. The current AI era allows OEMs like HPE and Dell to innovate in areas like cooling, power efficiency, and space utilization, unlike the PC era where differentiation was minimal.
  3. Nvidia’s inability to enforce a single reference design on the ecosystem has fostered greater innovation and partnership with ODMs and OEMs.
  4. NeoClouds and enterprises are increasingly seeking customized, efficient solutions to maximize compute per megawatt, driving demand for semi-custom systems from Dell and HPE.
  5. On-prem AI factories are emerging as a viable market, with enterprises buying a few racks for sovereign or latency-sensitive workloads, potentially adding billions in server spending.

Summary:

In this episode of The Circuit, Ben Behren and Jay Goldberg discuss insights from HPE Discover, focusing on the engineering differentiation in AI infrastructure. Behren highlights HPE’s hybrid cooling pod, which mixes air and liquid cooling to fill a customer gap, and ultra-thin liquid-cooled trays that pack more Vera Rubin GPUs per rack, maximizing compute per unit space. This contrasts with the PC era where OEMs had little room to innovate.

The hosts note that Nvidia’s inability to impose a uniform reference design has allowed ODMs and OEMs to engineer unique solutions, particularly in cooling and power management, benefiting NeoClouds and enterprises. They debate the role of NeoClouds as a middle market between hyperscalers and long-tail enterprises, with Dell and HPE balancing semi-custom designs to meet demand while managing margins. The conversation also explores the growing trend of on-prem AI factories, where enterprises buy a handful of racks for sovereign or latency-sensitive workloads, potentially revitalizing the on-prem server market.

Overall, the hosts see this as an engineer’s dream era, with intense focus on efficiency and differentiation driving value for customers and vendors alike.

FAQs

He observed that companies are heavily engineering AI infrastructure racks with unique designs, like HPE's hybrid air and liquid cooling, to differentiate performance, efficiency, and space usage.

It mixes air and liquid cooling in a four-rack pod, using a rear heat exchanger and a small closed-loop liquid system per rack, cooling more watts than air-only but less than pure liquid.

Unlike the PC era where designs were similar, AI infrastructure allows OEMs like HPE and Dell to innovate in cooling and compute density, giving customers advantages in compute tokens per watt and space efficiency.

They slimmed the trays to pack more Vera Rubin GPUs and CPUs per rack, optimizing for space and liquid cooling, achieving higher compute density per unit of space.

Nvidia initially pushed reference designs, but now fosters differentiation by allowing ODM partners more room to innovate, especially in cooling and system design.

NeoClouds need semi-custom designs to maximize revenue per megawatt, and they lack hyperscalers' resources to build entirely from scratch, making them a high-margin, growing market for OEMs.

Chat with AI

Loading...

Pro features

Go deeper with this episode

Unlock creator-grade tools that turn any transcript into show notes and subtitle files.