Speaker 1Hello, everyone. Welcome to another episode of The Circuit. I am Ben Beharin.
Speaker 2Greetings, programs. I am Jay Goldberg.
Speaker 1Well, this was a double event week combined with some juicy earnings and a lot of busyness on both of our parts. This week had Semicon West in San Francisco and OCP in San Jose, which next year, Semicon West is going back to Arizona, and OCP is going to San Francisco to be in the— On different weeks. Thankfully, different weeks. Thank you. Yes, yes. So we can all be insane in now multiple weeks instead of one giant insane week.
Speaker 2Exactly. There were also three analyst days or three analyst days-ish buried in there, too. Like, it was an impossible week.
Speaker 1Impossible week is an understatement. But we were there, so we're going to lump this into a couple categories. We're going to talk about Semicap slash Wafer Equipment, which kind of will combine ASML, earnings, TSMC, conversations we had at Applied Materials, and their Investor's Day. And then we'll talk about OCP on the back of that and the major themes that we saw. So, you know, one thing that stood out to me, and I've sort of been saying this fairly frequently for— in fact, we've titled multiple episodes this, but I do find the Semicap part of this cycle very interesting because when you listen to those making the goods or building the clean room, you hear a very interesting sentence. You hear them talking about what's going on in new equipment, where innovation is needed, where that wasn't necessarily the innovation pipe for tooling and machining before. We'll just say in the kind of monolithic designing a mobile and a laptop type chip that as we push radical limits and try to develop these giant AI accelerators and all of the things, including much larger—getting and getting larger networking ASICs, so let's not forget that networking ASICs are going to have a whole lot of advanced packaging on them and be bigger, not just GPU, XPUs. You hear this interesting theme from Semicap saying, you know, we're seeing the challenges in scaling this industry to meet the demands of those who want to use our fabs as fabless or use our tools in a fabless semiconductor. You hear this interesting theme from Semicap saying, you know, we're seeing the challenges in scaling this industry to meet the demands of those who want to use our fabs as fabs. For all things considered, I find that an interesting observation and not really one we've seen in some time. Perhaps the only historical precedent is when we were chasing Moore's Law regularly and doing node shrinks on an 18-month basis. You know, material and testing and lithography was a big part of that. And now it's like it's back in full force and everybody's talking about having to go and innovate, solve problems, and even do new machines that they didn't have to think about before.
Speaker 2Cereals had the grand opening of their r&d center their epic center in santa clara ben and i were both there um it was a very interesting event they had a lot of very high profile people i got the the the ceo of hynix was speaking and i got a phone call while i was there it was an important phone call that i had to take and i was afraid to leave the room because like i don't want to i don't want to anger the memory gods right so i waited and called the person back but it was like very high profile people applied materials it was uh very very cool that we could we could go there i would just have one sort of constructive criticism is more tools like we would love to see more more tools um than we're on display it would be like to me that's like seeing i would love to see a room filled with all the tools and robots running around and that would be that'd be fun so next year maybe so you know there's a lot happening in the space i think the other the other thing is tsmc reported this week and i i've i've given up trying to track what expectations were i guess it wasn't what people thought i don't know numbers look fine to me um but i they they said a few interesting things on their call right they they talked about terrafab too but they kind of played it down uh i i thought the most fascinating thing they said is they actually have their own employees going on site to data centers now around probably the u.s maybe around the world checking to see how far along these fabs are which i thought was very fascinating right here's a here's a foundry they are multiple layers removed from data centers but they actually have people doing site inspection uh trying to figure out what's what's real what's going to get built on what time frame which is is a is a big change from the world of the past right i remember the days not so long ago when people were like what's your forecast for handsets what's your forecast for pcs and that would be their modeling they're they're much more engaged now as i think they have to be and trying to figure it all out and then everything else i think from their earnings is pretty much as expected right yeah all the timelines are the same no big changes there but they're you know it's tsmc so it's good to see that they're chugging along yeah i thought
Speaker 1the data center point to be honest with you was probably the most interesting but but as a broad for tsmc i just again right i i'm trying to ben's not in the market ben doesn't recommend stocks or try to really analyze the stock market um i'm trying to keep these things separated because tsmc is the is the one company i mean i could argue there's many but i would just say historically how this company operates and its management operates this is a very conservative company and this is why as a bellwether having them go and say the things they're saying that they are talking to customers they're in the market they're doing these data center checks to make sure that we have the power to energize the gigawatts that they're going to make and sell and they're still investing they didn't you know announce a capex increase but people assume that they will before the end of the year or in the early quarter in order to build more factories more clean room space like they just this is a company that cannot risk having factories sit empty or underutilized we've talked about this before so the fact that they're doing this like to me has to be the most bullish cycle for demand because they would not risk it they just that's not how they operate and again i think that plus my points on semi-cap just you you have to be careful of the semiconductor industry being built on the back of what people are convinced and have deep conviction is is a growing demand profile not a overbuilding profile i guess is my is my point okay i'm not i'm not quite there yet
Speaker 2but we can debate that on a different day i uh in terms of demand there's a lot of yeah i'll save it for another time um but i i certainly agree there's in terms of demand there's a lot of demand there's a lot of demand there's a lot of a lot of there is a lot of demand right now it will stick around for a while um and we we need we we're you know we've i think i think the way i look at is we under invested we've the industry under invested over the last call it five years and now we're being we're playing catch up at the very least and have to get someplace very new i think the other the other interesting thing from tsmc was they they said that they're converting n5 fabs to n3 right and we touched on this a little bit last quarter i know ben ben thompson on stratechery talked about this a lot that's a that's a new thing for them usually they would just keep all the n5 n5 fabs running and then build new n3 fabs and they're actually retrofitting the n5 to n3 something that's new for them three nanometers in big demand i guess it's going to be that way for for a while going forward
Speaker 1yeah which raises the question i think i think we've either asked or noodled about or discussed in private uh what happens to all of the demand for five nanometer and below because it is still there there is still demand for let's just say trailing ish edge and as tsmc does this that creates opportunities for others likely global foundries who again is raising capex and moving to new nodes and benefiting from this cycle that uh once wrote off is now seizing an opportunity as well i i think the if we if we
Speaker 2talk about the trailing edge mature processes that's a different conversation there's a lot of a lot of factors pulling on that market um again topic big a bigger topic for a different day but i think trailing trailing edges is going to be a less less exciting oh for sure right i but it's still needed yeah it's it demand is still very strong there's an awful lot of that capacity coming on stream everywhere especially in china and how that all works out is still to be determined right and certainly that wasn't that wasn't like you know for all of the new fabs coming online in china nobody really talked that much about trailing edge at semi-con i think people wanted to talk about yeah of course new new things i will say there was
Speaker 1one i will say there's go ahead i was gonna say that's what drive new new new leading edge or new nodes is what drives new new equipment so it's like that's kind of why i think semi-con was talks about it's good for equipment makers
Speaker 2but but i've been reading up about this like uh trailing edge equipment is in in shortage too now in fact in china there's a very healthy secondhand semi-cap tool market and i think that's a very healthy secondhand tool market and it's got that has gotten so tight that people are stealing equipment right you know you're a broker you you sell used used edge tools you have a warehouse somewhere somebody pulls up a truck oh my gosh presumably with 10 guys and a forklift and steals equipment like that's how bad how much demand the organized
Speaker 1crime has moved from robbing banks and other things to robbing semiconductor equipment yeah that is the day the day we live in yeah um yeah i interesting stuff i mean for sure um but like i said i i think there's a lot to noodle on around the shape of the semiconductor industry post 28 29 on the back of all of this additional capacity and new tooling um innovation and process and radical limits and packaging and all the those things it's going to be super interesting um to see
Speaker 2i think there's there's a couple other things i i picked up and i think are worth mentioning one is asml is apparently gonna have a new tool to do hybrid bonding right and which is interesting because i we usually think of asml as strictly a lithography company but they have other products and they're getting into hybrid bonding i've i haven't dug too deeply into this um just haven't had time but my sense is that they have a ways to go um it took you know it took bezzy 10 years amat took 10 years to get to where they are so i think that there's there's still a little bit of learning there but it's interesting that asml is branching out in that way certainly hybrid bonding is is is here to stay and give me more and more important but that also leads to this sort of interesting debate that's taking place in the memory world um which is how high do you want to stack hbm and this is like the one place like there there are a few uh like i said applied materials had a analyst event and they had a a breakfast for broad audience and they had a separate q a for the sell side and digging into those weeds this is this is the one place where i see a little bit of light in between my view and what applied is saying because they they seem to be very bullish on this idea that we're going to hybrid bond hbm wafers to each other to get increasingly high which is very, very different than what we heard at Hot Chips back in August, where people kept saying, no, no, no, don't do that. And when I say people, I mean end customers who are buying HPM. They're saying, don't do that, don't stack it higher, just give us more of it. And Applied seems pretty certain that we're going to get stacked HPM. I don't think I agree, but they have good reasons for thinking it's real.
Speaker 1I'll comment on this only as an observation that is a let's watch for this because a question did come up from our friend Vivek Arya on this of B of A and he basically asked them this point, right? What's going on with 12 high, 8 high, 4, et cetera? And they made a comment that I think is worth watching and thinking about, which was essentially that it depended on the workload, that there are workloads and they erred toward a training workload, which has and can, I guess, better utilize higher stacks of memory for its advantage. And then there's other workloads, call them light inference workloads or maybe medium reasoning workloads that while they could use 12 high, are not going to use it at capacity effectively for, you know, call it sustained use. So I don't know the answer to this because I think there's all sorts of variables. But what I would say is, I think this is a model point. Like where models go can either necessitate higher stacks of memory or not. Like that's a software, the software has to take advantage of it. If it can't take advantage of it, then it doesn't make any sense. If it can take advantage of it, then I can see, right, a point. Now, again, is that every workload? Is it 20% of workloads that can take usage of 12 or more? Is it more or less like that? We don't know. But I think that was kind of how they answered that. I've heard others deepen the weeds in this, say a similar point. So I would just say, dig into that and keep an eye on the workload. And whether the workload can demand a higher stack of memory or not as a greater proportion of how it's used on a utilized on a daily basis by the masses. Okay. I'll, you know, we'll, we'll, we'll see. Before we go into OCP, I wanted to share an anecdote that I shared with you and I want to share with everybody. I witnessed a fairly fascinating event in San Diego this weekend as I was walking in the neighborhood with, with my child who goes to school there. You know, any, you know, as any good technologist, medical person or analyst does, right? You walk past somebody chatting and they say a couple of buzzwords that catches your ear and you, you hone into it. And so I heard data center get talked about by these two gentlemen who were having a conversation and one guy was pretty upset about this. And the other one was kind of talking reason and trying to talk about the benefits of data center, having a good, a calm conversation, but clearly some emotion involved. And, and I turned around and just kind of looked and the guy who was, you know, in. In the, you know, pro data center camp had all these pamphlets and it hit me that this person is evangelizing data centers door to door in this neighborhoods of San Diego. And I thought this was fascinating. I really wish I would have gone back and been like, you know, give me your story. Who's behind this? Like, is this the energy companies? Is AWS pay? Like who's paying you to do this? But I just like sign of the times again, right? Like, like. That weird, weird timeline we're in that there are apparently people door to door evangelizing the positives of data centers to communities because of the community backlash. So I thought that was, I thought that was very interesting and entertaining at the same time.
Speaker 2I wonder if the guy will still be doing it after the election.
Speaker 1I, I, I, it's a good point. I mean, I, I don't know. I mean, I think as long. As long as there's this kind of hostility, I can see these types of like evangelistic things trying to just help communities because where else are they going to hear? I mean, think about it. Think about you, but you're right. It feels like a election issue because this is the kind of thing you do to educate people who you want support for, for things like this is playbook one-on-one of how to change minds. And but it's just, it's just interesting to me that that's the tactic. Because again, where. Where are you going to hear it? You probably won't see a reasoned argument on Facebook or Instagram or the news. And so it's actually an interesting medium to try to have more informed conversations. So you're right. I don't know, but I found that a, an interesting tactic that's going on to attempt to soften communities around data centers. You know, the show picture door, it's not a religious person, right. Or, or somebody selling something. It's a, Hey, did you hear about how good data centers are? For, for our city and your logical future,
Speaker 2brother, can I give you the good word of Sam Altman of data
Speaker 3centers of our, of our intelligence future?
Speaker 1Okay. Anyway, I thought that was a fun anecdote. Okay. Uh, and I, and no one, no one in my family is in technology. So I was the only one that appreciated this situation, but there you go.
Speaker 3Okay. Let's go. That's why you have a podcast. That's right. That's right. That's why I texted you immediately. Okay. All right. Uh, OCP. I, I actually really like.
Speaker 2Wait, wait, before, before you start talking about the show, can I just, can I just flex a little bit? Please go for it. Day, day two of OCP. I got street parking. Really? Yeah. Huh? It was, you know, it was a bad, it was so, it was so crazy.
Speaker 1I mean, I, I, just to even give you more of us for anyone who's been involved in event, you know, you can sort of judge how hard it is to get an Uber by like the density of something. And I had to wait 25 minutes to get an Uber. In fact, I didn't think I was going to get one. I was like, this guy doesn't want to drive me 15 miles away because he wants to just, they just want to do these little short drives. Like I'm, I'm going to be stranded here and have to call a family member. To come pick me up from downtown San Jose. And that was how busy it was. It was, it was, it was GTC level busy, which was pretty interesting. Yeah. Not quite, but, but pretty close. And GTC was almost 30,000 people. So, um, all right. So I actually really like this show, um, for a lot of different reasons. You know, the open compute project is really something that is standards based, which I think is good. So like you talk to. A lot of people on the floor, you talk about rack design, you talk about cohesiveness and they're talking about, you know, that this isn't an OCP spec. And so it makes sense, right? You want an industry to have, uh, you know, co cohesivity across it's, uh, it's it's ecosystem for, um, for support and growth and standards, even though there's no standards in optical yet, but we'll get, we will probably get there. Um, but I liked like having these kinds of conversations, especially around some interesting pockets. Some around networking, because again, there was a lot of what's going on in ethernet, what's going up, what's going on in, um, different types of, of networking that is standardized as well as what was going on in liquid cooling, which I thought was super interesting. Cause a hundred percent that actually does need more ubiquity than it has today. And then the 800 VDC conversation that was starting to come up with again, pockets of fragmentation, but, but these things need to be standardized so data centers can install them operate. You know, all, all of the things, but I like, again, right. My point is this show is around people trying to build together on standards on a, on a baseline way of doing things so that there is interoperability as far as that can go one way or another with differentiation. So that's like the center of it. And when you would talk to people and their themes of like, yeah, it's OCP compliant, or we're working with OCP to make this more of a standard for things like. So that's like the center of it, but I like, again, it's OCP compliant, or we're working with OCP to make this more of a standard for things like that. The one I thought was most interesting, which is the shift from copper to optical backplanes, which once you go to optical in Iraq, whether that's compute or ASIC, uh, you're not going to have copper on the backside of that. You just, you shouldn't, uh, you're going to say you want to slide this tray in with an optical backplane. It has to align up once you slide it in. And so these, these were interesting discussions talk about future rack design for scale up and scale out, uh, type of type of networking compute domain. So that was just, just one example of. Rack compliance size standardization, right? The way in which you plug things in, um, anyway, that's extending now also to liquid cooling and an 800 VDC, which is positive, positive for the industry.
Speaker 2I, so just so everyone knows we're recording this late at night, so, and it's been a very long week, a very long weekend for me. So I'm a little extra. crunchy um spicy i liked Spicy. I liked OCP. I thought it was a great conference. I think it was very good to attend. I know some people were frustrated that there wasn't a lot of news, but just the attendance and just the quality of attendees was really, really good. I would just be standing in a booth for whatever random company and the CEO would show up and start talking to people. Not investor meetings, just like he was coming into either his booth or some other booth to talk to somebody. It wasn't a pure marketing fluff event like some conferences have become. There was a lot of substance there and a lot of really good people there who knew what was going on. You could go into a booth and ask a technical question and get it. It wasn't a serious technical answer from most of the companies, which I thought was very useful and very important. I'm not a big believer in OCP ever becoming a standard. Most people I talk to tend to sort of – they view OCP as being useful for other things, but I don't think anyone ever really adheres to the standard itself. That comes out of nothing. That comes out of OCP. It's a good guideline. My example for this is OCP really started off by standardizing rack design. That was an important thing where everyone is like, this is how wide the racks are going to be. This is what a U is. We had some of that before, but this is kind of – there were some – they weren't really standards. They were just common patterns that people adopted. But then the number of companies I talked to who had these really weird-sized rack trays, it was mind-boggling because we're now moving to – You know, NVIDIA, Kyber, and AMD Helios are double-wide racks, and people are starting to say, oh, well, if we can double the width, let's do this. And so I started to see server trays that were sort of one-and-a-half times the server tray width or two-thirds of – like all these weird configurations started to show up. And there was a Chinese company that makes trays, and they're like, you just give us the measurements. We'll make it whatever size you want, which is – right?
Speaker 1Yeah, and in the grand scheme of, yes, there are a handful of companies who can do that. The other example that you just – that we talked about, so the Microsoft Maya tray was a different-sized tray. Those who have their ability to do those things, yes, but then they're still in those rack sizes, whether that's the connectors or the ports or whatnot. Like there's still some – you've got to be able to scale these things, right? You've got to be able to meet demand. So there's some level of – Yeah, that's fair. You're designed to a spec. And then there's other parts where you've got – you're right, flexibility for those who have. But something, again, like I think 800 VDC and to some degree, right, direct chip liquid cooling, which you talked to everybody who was there, were talking about the need to co-design the liquid cooling plate to the chip design, which is true, right? You've got different sizes. If we're going to push reticle limits, right, it's got to push sizes. You're designed to that spec size. Case in point, a lot of those – Case in point, a lot of those examples that you saw in a liquid cooling-sized template was a Broadcom Tomahawk 6 chip. It was the exact same size because they are building to the spec to cool – liquid cool Broadcom Tomahawks, right? So it's like – it's just things like that that the industry needs to scale. It needs to design to a spec. And so there's areas where you do that, and then there's areas where there's flexibility. So I agree with that. I will say, though, I have one concern. And, you know, Ben's going to play – I don't want to say I'm a pessimist, but something. Think about what we're literally on the back of right now, right? You're talking about the future of the data center, and you go, yeah, it's going to be direct-to-chip, single-phase liquid cooling, designed to spec. It's going to be – which includes these insane manifolds, which I took pictures. You should see on Twitters. And it's literally – It looks like a car engine to a degree, right, that's cooling anywhere from a couple hundred megawatts to – or a couple hundred kilowatts to the largest one from Delta, three megawatts, the largest – which was a huge, giant cabinet. But you talk about the future of the data center, and you go, it's going to be optical. It's going to be – like all optical. It's going to be liquid cooled, and then it's going to be 100 VDC. I don't see how anybody in the next couple of years can do all of those transitions at one time. Like you're going to need to pick your transition. Which one am I doing? Am I starting with liquid cooling? Because that's already an entirely new design. Am I going to then figure out how to add optical in there? Again, another design. And then I'm going to throw in these sidecars and have to do my step change of 800 VDC. Another transition. It just seems very unbelievable. For everybody who's like, look how big this market's going to be for all three of these things, that those transitions happen at the same time. That three – those three fundamental transitions happen at the same time. I just don't see the reality of that.
Speaker 2So let me start off by saying it was very noticeable that consistently the most crowded booth at the show for all two, three days was the Delta Electronics booth. Which is crazy. Confusing to me. But like it makes sense in the context of that show. Everybody wanted to see this just mammoth cooling system. But yeah, did you notice that like that was the most crowded booth?
Speaker 1I got elbowed in the chest like multiple times. I had to throw elbows to get anywhere near this thing.
Speaker 2It was packed the entire time. I've seen other booths that are crowded, but usually that's at like 530 and they're serving free alcohol. This booth was packed from the beginning of the day until the end. Everybody wanted to look. Everybody wanted to look at it. I think though that your point about timing is actually my – I walked away from the show. That was the one sort of biggest concern I had on all of this is just how long some of this is going to take. Co-packaged optics is going to take a while. 800-volt data center is going to take a while. Cooling is – all these big liquid cooling systems are going to take a while. It is – it's a lot. I think you're right. There are going to absolutely be people who are going to pick and choose. We'll do one, the other. I mean the problem is some of them you have to do all at once. Like how do you have an 800-volt data center that's not liquid cooled? You need – same thing with co-packed optics. But maybe it doesn't need to
Speaker 1be all optical, right, for example.
Speaker 2Yeah. So I think that is partly why Delta was so crowded is I think most people were realizing they're going to have to go – if anything is going to get accelerated, it has to be the liquid cooling stuff and all the other things are going to be contingent upon that and have to wait. But I definitely – like I spent a lot of my show talking to people about 800-volt data center. I spent a lot of time in TI's booth and in ADI's booth and looking through all their parts and in both booths I had to – I got elbowed by someone who was almost certainly a spy like for another foreign country. Same. A hundred percent like – Spies were around. There were a lot of spies around. Yeah. And – Yeah. There was – for me it was interesting because I saw for the first time like real products. One of the companies I talked to had – was explicitly saying that they had a design win with an NVIDIA system. They all have this sort of part – all these analog companies have this part that fits. It's very small for – it's a lot of chips, this system. It's a lot of chips and they – but they put them on a very, very small board. And that's so it can fit in a compute tray. And these guys were – pretty clearly had won some kind of design, more than a qualification. I think they actually were starting to talk about volume for their part in NVIDIA systems in the future. But like timing and all that is 27 a little bit and really it's 28 for 800-volt data center. It's big. It's coming. It's just going to – I think we will very likely see growing pains where people start doing it and then they realize, oh, we didn't think about this and we're having faults on that and we have to slow it down. We're going to have some fits and starts here.
Speaker 1I want to come back to the size of these things because what that got me thinking about, which is actually a networking point, when you talk to everybody about the reach of copper slash optical, what's going on east-west and you tell them to talk about – Yeah. Yeah. We're going to go up to much faster speeds, needing to go east to west. But picture that distance conversation is largely being had without a three to four-foot wide manifold cooling system sitting in that rack, in that row. So you're sort of like, okay, yes, you want to cluster four or five or how many of these things? But you're also in row going to have a very large cabinet and then another one when you want to do an 800 VDC sidecar. Like, okay, you're running out of space for all of these racks. They take up more space. So it's sort of like if – data center space is the equivalent of clean room space and you're trying to get your dollar per square foot you have just introduced two pretty large things that that solely exists to keep it cool so you can again crank more tokens per watt out of that thing but you're losing rack space by adding these gigantic manifold coolers and 800 bdc sidecars to the equation of your rack row
Speaker 2i'm picturing somebody two years from now saying hey we we decided to go with copper we we use all this fancy electric active electric copper because we thought you know we needed that extra distance we need that cable that could go two meters but we forgot about the sidecar we actually need three meters and so and so it turns out we need optical after all yes yes that's somebody that's gonna happen somebody's gonna be like oh we forgot about the sidecar and
Speaker 1we gotta well and and again as cool as that delta three gigawatt one was that was probably eight feet wide like that's not a small cabinet to be in a in a row and sure the as you can see over here my point though the benefit is one of these that can cool more means you need less of them so i get the benefit it's just it's one of those things that you have to consider in the grand scheme of your square foot as you add these very large manifolds so again hence you need networking it's going to add distance to your compute pods and so you've got to deal with all of these things you know the other thing i i heard from a lot of the vendors on just again a point about ocp was uh maybe a little bit more i mean i don't want to say this doesn't happen at computex and probably the best way anybody should think about the show is kind of like a mini computex not quite as big but a lot of the same kind of cool stuff thematically vendors but i heard from the vendors that there was a lot more deal making happening at this show that they were engaged with customers having real engaged ordering types of decisions like what do you need um that wasn't necessarily say the same kind of case at at at computex so i got a ton of value from this um and i thought again like i i heard i was i won't say who um but i was at a booth for a very popular technology doing cpo and i saw one of the largest oems data center uh uh rack oems basically say we need to talk let's let's have a conversation we need to do
Speaker 3a deal i'm like hey okay my
Speaker 1snooping ears win again um but but that's that's what was happening here like people were going and seeing things being yeah i didn't know that existed like let's let's have a conversation i probably want to use this in my technology or in my solutions
Speaker 2i'm not sure how much you anonymize that but but yeah no it was a real show like there was there was a lot there was i'm not surprised to hear that there's jail dimming like because all the all the principles were there yeah right and i don't know i i still haven't gone through the sessions the technical sessions i haven't either i made it a point not to attend any because they were all the videos will be online um but the you know the i keep hearing the ones that were most were most crowded were uh cooling and copper everybody want to talk about copper and i think that was to me the sort of big mystery that came out of the show was um right there's this just as a refresher for everybody optical is coming because copper we're reaching the limits of copper and the problem is every we've said this before we've had this conversation before we're like oh that's it that's it for copper it's time to go to optical and every single time somebody pulls out another trick to make copper go a little bit further last a little bit longer um there's a very strong preference for using copper it's just much simpler and i think there were a few of those at the show too from what i can tell they're not perfect solutions they get pretty complicated pretty quickly so you don't really have this it's not really true copper anymore um but i think to me that was the big mystery was this debate about sort of the timing and sequencing of how much further can copper go can we push it another generation when does co-package optics available are we going to do something in between like any npo first i i thought i was going to walk away with the show from the show with a very clear answer on that and i think it was the opposite it just got much blurrier for me yeah although i did think of a really good analogy for copper versus optical can i share it okay let's say you got some money and you want to buy a nice car you can buy you can buy a brand new call it alfa romeo it goes really fast it's beautiful it's the newest thing right but it you know it has a lot of electronics built in it's very complicated you don't know how reliable it's going to be you could also instead of buying the fancy new alfa romeo you could buy a classic 68 mustang right and you know that that mustang you can fix it like you can get on youtube and figure out how to fix it yourself you have all the parts for it it's very straightforward it's not going to last forever but you it's still you can get a lot of you can go pretty fast it's not going to go as fast as the alfa romeo but it's going to go pretty fast right so that's copper versus optical take it a step further you got a lot of money and set in your fast car you're going to buy the ferrari right you can buy you can buy a ferrari it's going to go really really really fast much faster than the than the mustang could go right it's but it's very complicated it's hard to buy there's only there's only one way to buy it um so that's co-packaged optics right you've you've heard reports that it's actually much more reliable but you don't know that and you know that like fancy modern sports cars aren't super reliable you're you saw on the facebook post that it's no no these new ferraris are super reliable you don't know if you can trust it right that's co-packaged optics yeah
Speaker 1that's my metaphor yeah and i think when you see these in person to be honest with you it's it's hard to tell if it's a ferrari or not i don't know if it's a ferrari or not i don't know if it's a ferrari or not i don't know if it's a ferrari or not i don't know if it's a tomahawk solution right that shows this it's still um serviceable right you can still clip the lasers on but it's basically over it's it's on top of the edge of that surface area and then same for a compute it you look at it in your you like you realize how much more challenging that is than any form of it's copper alternatives in in complexity and how how it sits on that chip like it's it's it's it's a head scratcher honestly every time you look at it as to like how are you really going to scale this but but again we we we're pretty sure we'll get there at this point i don't know how much i'm going to say like first people are trying to solve hard problems um we will get there we'll get there
Speaker 2because eventually the mustang will wear out just can't last forever eventually we will get there yeah but let me
Speaker 1let me make a contrarian point on this though like let's just say that copper can get to i think what is it right people have confidence in 800 gig and i think they're i think they're talking about it still in 1.6 um but but let's say that this these these compute domains we talk about um makes actually take off and you've you're at now able to have three four or five racks together in parallel which is really the main reason why anybody is trying to do compute density in a rack so you can run a bigger model than you can run today in just that one rack because that's largely how this works it's served and inferenced from one one racks compute and so you can now bundle four or five of those together well why why if that's the case would i want to increase my gpu density in one rack at which point maybe copper is going to be okay and so you can now bundle four or five of those together well maybe copper is going to be okay and so you can now bundle in north to south for much longer as you can connect scale up pods in east to west and run those those those larger models across three or four racks instead of trying to do it in just one and if if that's the case i think you and broadcom actually has a very strong point of view here about how long they see tomahawk being able to increase with their 30s to do copper in a rack and you'll still get more jet embed city than you have today so you could make a lot of money maybe go to 500 gpus um but even you know you're doing theoretically you're doing 144 today too because right that's that's 72 is 72 packages of 144 gpus and so you know say that you you can scale that a little bit more and now you're again you're you're you're networking all these racks together you you could make an argument that maybe i don't need that much density in one rack and then therefore maybe copper is going to be okay and so you can scale that a little bit more and now you're you could make an argument that maybe copper is going to be okay and so you can scale that a little bit more stays around longer i always like
Speaker 2a good contrarian argument um i i i see your point i i think though they're just getting so big right i mean people are talking about you know million million node worlds that's on people's roadmaps i just i just don't see how you do that without optical right it just gets it gets yes yes yes but that's east
Speaker 1to west i'm saying like what if the assumption was copper goes away north to south what i'm saying is what if this scenario is you get that density by these fully connected nodes and therefore you can keep there just becomes a reasonable limit of how many gpus you have in one rack and you can actually maybe still do that in the true scale up north to south domain and then you connect that to the one next to it right with optical which is what i'm saying is what if you get that density by these fully connected nodes and going on today but that's a bigger brain now it's not just limited to the gpus per rack
Speaker 2don't you think the gpus themselves are getting so big and so fast that the data rates overwhelm copper but that's
Speaker 1the theory but again the thesis for that was i want to pack as many gpus into one rack so i can serve a large model per one rack my point is if you could do that on two or three and four racks easier with more reliability and you're increasingly scaling up those domains maybe the argument is not because think about like we have this conversation about kyber nvidia didn't show kyber so people are like oh what's happening to kyber does it have the same problems you are really adding a whole nother level of complexity when you start packing that many gpus into one rack and so i'm just saying what if the engineering trade-offs aren't worth it just network two or three or four of these together and you get the same thing and then you can let you you can have these engineering problems not be so hard as trying to put 500 something or more gpus in just one rack
Speaker 2i i like that because it it tees in nicely with what we were saying before is you you can't do all these things at once you're gonna trip so maybe that's maybe that's the way it goes this is the next is is copper stays around for north south a little bit longer right maybe maybe although although i will say the other the other big news that came up from the show uh was uh nvidia talked about network they had a networking session and they showed their ruben ultra yeah deck pod and they said yeah pod yeah and they said that would that would have scale up co-packaged optics yeah and so that that brings everything forward because that would be next year yeah right ruben ultra is next year most people i'd assume that co-packaged optics wouldn't come until fineman which is in 2028 and so nvidia seems to be hinting that it will come sooner than that now personally i don't think that's going to be the volume networking stack i i don't it's not possible like there won't be enough parts yeah right it's a mix it's it's one they're gonna i think it's sort of like they're putting their toe in the water they're saying it's gonna be capable of it but i don't think many people will actually deploy it just for no other reason that you just won't be able to buy them yeah
Speaker 1yeah absolutely um
Speaker 2but this does it does sort of lay to rest there was a big debate earlier this nvidia gonna go with npo for ruben
Speaker 1ultra yeah um and so but again that's the networking switch and that's still east to west not yeah yeah yeah no i know but
Speaker 2they're still they're still
Speaker 1yeah in fact if they if but if they pull that off i actually think it makes my point stronger that you just put a massive pod together through cpo switches east to west and you left copper north to south and you now have a giant scale-up domain that's better than my conspiracy theory
Speaker 2because i can't share my conspiracy theory on on air in a public forum okay
Speaker 3i'll tell you about it afterwards but it's not
Speaker 2going to make you feel good about the world okay great that's amazing uh all right
Speaker 1well then we're gonna leave you on that cliffhanger everyone and uh we look forward to talking to you next week have a great one and we'll talk to you later
Speaker 2thank you everybody tell your friends tell your agents click like subscribe thanks for listening
PodcastTranscript AI