Speaker 1In an AI digital age, energy capacity equals economic capacity. Why build a new data center and buy a million GPUs if I can't power that? You're going to see more and more defaults happening on many of those data center projects because the energy won't be there.
Speaker 2Whenever you have the technology to make something easy, that means the bottleneck moves somewhere else.
Speaker 1Nothing's a chip anymore, it's a rack. It took me three months to design it, but it's nine months until I can actually start to use it. Exactly how many major new memories have you had over the last 30 years? Zero. Memory innovation for the first time in 30 years is nigh upon us. I declared the death of copper about 25 years ago. Eventually, I'll be right for all of us hardware guys. It's like renaissance in front of you. AI inference accelerator chips, and I'm sure I don't even know them all. Why are you guys funding so many of those anyways?
Speaker 3You've funded your first share too.
Speaker 2Historically, there have not been 100 competing processor vendors in any industry. Ever. Is this a temporary thing? It'll convert back to a few? I see it as.
Speaker 4AI is making it faster to design chips. But actually building them, powering them, and connecting them may be getting harder. In this episode, A16Z's Ragu Raghuram and Guido Abenzeller sit down with Pat Gelsinger, general partner at Playground Global, and former CEO of Intel and VMware, to discuss the next bottlenecks in AI infrastructure. Pat takes us back to his early days designing Intel's process, when engineers were inventing the tools they needed to build the chips themselves. He explains why AI could bring a similar shift today, even as manufacturing timelines, memory bandwidth, and power become harder constraints. They also debate whether the explosion of specialized AI chips will last, why Pat believes memory innovation is finally approaching a breakthrough, and how optical networking could change the architecture of AI clusters. In drawing on their shared history at VMware, they close with a question, "What would it look like to rebuild virtualization for a world where AI agents, rather than humans, are the primary users?"
Speaker 3I'm here with our esteemed guest and dear friend and former boss, Mr. Pat Gelsinger. Welcome, Pat. Pat is currently the general partner at Playground Global, but as you are very, very well known in the industry for leading Intel, for being the CTO of Intel, and of course, leading VMware and many other things.
Speaker 1So, welcome. Hey, thank you, Raghu. Great to be with you and Guido. Right, and to me, this feels just a little bit like old home, right? Yeah. We've superimposed a year of change, right? But you both look the same. I feel energetic, so let's dive in.
Speaker 3There has never been a day when you stopped being energetic, so there's no news there. But yeah, no, let's start actually from your Intel career. Recently, Andreessen Horowitz created this Horowitz Andreessen Academy, Mm-hmm. which was. was to find talented people between 16 and 22 and then put them into a modern educational setting so they'd be well-prepared for either doing things on their own or joining the companies of today. You had a similar experience, like you went to a regular trade school and then jumped into Intel Yeah. when you were, what, 18 or 19, I should say?
Speaker 1Yeah, it was really a sort of magical period. I'm 16 years old, I accidentally win a scholarship, I go to tech school, right, I skipped my last year and a half of high school, and Intel comes recruiting when I'm 18 years old. And I'd never been on an airplane, had already fallen in love with computers at tech school, and the interviewer, right, Ron Smith was his name, he writes on his page, and I was number 12 that he interviewed that day, and if you've interviewed 12 people in a row, you can't tell male from female. Yeah. by the end of it, right? On number 12, he says, "Smart, aggressive, arrogant, he'll fit right in," so I got invited to. There you go. come to Intel. to Intel, and it just became really glorious, starting as a technician, moved into the design team at the end of the 286, engineer number four on the 386, architect and design manager for the 486, all of that while doing my bachelor's, my master's, and Ph.D. work, so it was like the best career that you could possibly have. You're learning by day, and you're putting it to practice at night, and.
Speaker 3That's right, that's right, they're creating new, completely new era of chips for. I bet most of what you learned and what you built, there was a big gap, because you're breaking new ground.
Speaker 1Yeah, in a lot of ways, and it really was a pretty incredible time in the industry, and I remember one of the professor classes, he postulated a new carry-look-ahead adder design, right, in one of my classes at Stanford, so I said, "Well, I'm working on the 386 carry-look-ahead adder, so let me go try it," so I come back two days later to the professor, and I said, "It doesn't work," right, and I started an argument with my professor. My professor at Stanford, you know, it was like, what, the next two class periods were just me and the professor arguing over why his design didn't work. He had a book that was being published with that design in it, so it was very annoying. Oh, it was very time-sensitive for him. And we started built-in self-test, right, and I remember Ed McCluskey at Stanford was my professor, and he was sort of the father of that, and we had wonderful arguments around why most of his ideas were bad, they weren't practical for real chips, but I was putting built-in self-test to work, in the 386, and John Hennessy was my thesis advisor, and obviously he did okay, right, I said, "I helped his career, he became president, and he's done okay since then," so it was just.
Speaker 2Later on, we'll talk about the CIS courses, this kind of debate. I had the same experience when I was teaching at Stanford. You come with something that you think is the truth, and then the student comes that works in industries, well, actually, not quite, so the next lecture, you come back with, "Okay, I looked this up, you're right, this is outdated."
Speaker 1Yeah, and I'll tell you, you know, as part of the. Yeah, that interaction, and I think that's part of what makes Silicon Valley so unique, right, you know, is you have so many companies, and entrepreneurship, it's like in the water here, right, starting things up, and new ideas, and challenging the professors, and you can't be a professor at a place like Stanford and not expect to get challenged and enjoy it. There's a good foot traffic in and out of that ivory tower.
Speaker 3Yeah, 486 was your big accomplishment, if you will. Actually, lots of people worked on it, and you guys broke some new ground in chip design there, right, and increased what functions the chip was supposed to do, and so on and so forth.
Speaker 1Yeah, it was a pretty magic period, as well, because it was before what you would think of as the EDA industry.
Speaker 3Oh, wow, yes, that's right.
Speaker 1And at the end of the, a little bit of the 386, but the 486 was really the first chip to implement what you would think of as modern EDA techniques. We did a high-level design, right, a description, an RTL description, but there was no Verilog, so we invented HDL, right, the Intel Hardware Description Language, so I wrote my own language. Well, you had to build a compiler for that language, right, so we created a compiler for the language. There was no automatic placing route, so we had to invent that, and we worked with Alberto San Giovanni, Vincentelli at Berkeley, and some of his students for the first, you know, place where the first routing, the first automated. Yeah, yeah. Timing, management.
Speaker 2So they created EDA then, to some degree.
Speaker 1Yeah, to a great degree that, you know, and that was really one of the hallmarks that not only was the 486 this compatible pipeline microprocessor, but we ushered in many of the foundations of what became the modern EDA industry, and it was really a pretty magical period of the industry.
Speaker 2Yeah, yeah. If we look at today, will people look back as Blackwell and saying, this was of the last generation? So then before AI took over the design process for micros, are we in a similar transition right now? I think there are certainly
Speaker 1aspects of that, and I think when you look at things like Jalapeno today, right, you sort of say, that's sort of a first principles use of AI in the chip design process, right, where you throw out a lot of those things. Now, there are pieces of design that are still really hard, right, in that sense, and a lot of the analog, right, aspects, certies is probably the best example of that, you know, are still. Not AI-able, right, you just need lots of silicon data, right, to get those, so you either get so conservative in your analog design that you're able to, I'll say, AI it, or, right, you got to do those the hard way. But now many of the logic functions can really be done with AI tools and techniques in pretty incredible ways, right? Your transistor budgets are big enough, the design tools are getting smart enough, a few experts guiding the tools and how to apply it, and I really think it will be somewhat. Yeah. Like the 486 that way, right, we can sort of look back and say, yep, that was the beginning of a new era of chip design.
Speaker 2Yeah. Can we guess at this point where the bottleneck will be in the future? There's some things which it seems like AI is incredibly good at, right, like creating the software layers, you know, writing kernels, a lot of the, I think a lot of the sort of being able to specify the objectives at the highest level. Which then gets sort of translated into a very log, right? And so what is the new frontier? What is the thing that will be difficult going forward?
Speaker 1Well, when you look at today, so let's say Guido and Pat, we're off to do a great new chip together. Building the AI. Right, right. And it's the one that's going to leap ahead because we have understanding of the AI.
Speaker 3It's the 2036, it's the 2486.
Speaker 1We have understanding of the AI workloads, we have understanding then of how to compose that into the right multiply, accumulate, register structure. Perfect visibility with model workloads. - Yeah, and all of that kind of stuff. And let's say we turn our aim tools loose we are great within three months we have an awesome design right for that we've been a little bit conservative on all the analog components of it now we still have a couple of really major bottlenecks today one is right we still have nine months of silicon processing time yeah right so like that all right so i can design the thing in three months but i can't actually get it into real silicon at scale for nine months okay that sucks right so we have to really see that bottleneck improve if we're going to have this innovation because we know what the validation
Speaker 3and verification stages are even beyond
Speaker 1that well you know first i can't get anything out of fab you know in less than three months and that's if i have you know like supercharged design flows right so i said you know you know how do i compress that right uh you know for it and then i got to get into advanced packages 3d packages are getting complex so that's like another three months right you know and then i have to put it into a rack scale solution because nothing's a rack you know so we're nine months you know it took me three months to design it but it's nine months until i can actually start to use it you know so how do we start to compress you know those aspects of design and i think we need new forms of lithography you know to enable that we need you know design flows that don't require you know 50 million dollars of mass cost until i can get things into prototyping so you know to me how do you compress that to a month or two right because i can now do the design in three months how can i have that done in a month or two because if it takes me a year and a half until i actually get scale and software on it okay my understanding of the ai workloads is no longer applicable to the chip that i design right you know so you know all
Speaker 2of those that play out live right now i think with the agents yeah
Speaker 1right you know it's almost you know and we can look at it historically it was like you know graph core wasn't a bad design but the world moved on right you know and right you know uh in that so all of those aspects somewhat obviate the fact that ai has made the design piece easy all of those manufacturing scale issues to me become a huge bottleneck right another bottleneck of course is memory yep right you know memory bandwidth we noticed yeah right and you know for that as i've described hbm is a hideous memory it's just the best one that we're asking about that yes it's a terrible memory you know i get terrible bit density i'm limited to shoreline bandwidth the power right uh issues thermal issues d-ram doesn't like to be hot you know concentrating everything in one
Speaker 2spot yeah you know
Speaker 1it's terrible to give an ai is a memory compute workload yes right so right you know for that so i you know i need to you know compress that development uh and embodiment into real scale you know i need to fix memory right uh associated with it so to me that's another critical bottleneck for us to attack and uh power right you know power in you know thermal uh and most of the power simulation aspects today are bad right you know you don't have good 3d modeling tools for power power dissipation hot spots uh etc you know my voltage in right you know i'm essentially guard banding almost my full power envelope correct right so essentially i'm losing you know forty percent of my power and guard banding so i got to get much better at power modeling you know and then my power dissipation techniques are bad right so you know
Speaker 3every model has to change
Speaker 1too yeah so all of these things you know to me those are the you know the next big bottlenecks right you know how do i you know improve each one of those and now right you know if i've done that okay now we're on to an exciting era for what's next so you know thank you for those great ai tools but you know i got big challenges even when those are
Speaker 3fully deployed and so we've seen sort of a rebirth and the number of chips and chip architectures and so on and so forth yeah yeah thanks to these workloads yeah it's
Speaker 1like there's a hundred ai inference accelerator chips and i'm sure i don't even know them all yeah exactly yeah so so why are you guys funding so many of those anyways
Speaker 3you've wondered your fair share too yeah so um so i read of they're all to their credit making different design assumptions either in the components that you talked about or in the in the part of the parede curve that they're targeting so do you think there's room for all of these or are we in a world where partly because of what we talked about ai tooling um we now have the ability to build more different kind of chips yeah you know the time is enormous whichever way you want to look at this even more provocatively
Speaker 2historically there have not been a hundred competing processor vendors in any industry ever right so yeah yeah is this a more heterogeneous future will we see is this a temporary thing it'll convert back
Speaker 1to a few yeah and you know i see it as a more temporary thing and it will convert is what i expect and there's probably three different reasons that i see that to be the case you know one of them is you know this emerging heterogeneity that you see in the compute uh environment where okay now i have specialized pre-fill versus decode well you know and now people are saying well we really need to specialize tier for mid-fill not just pre-fill right you know so now i have early pre-fill and now i have mid-fill which is you know different maybe
Speaker 2speculate and verify if we
Speaker 1listen to open ai here right yeah you know and you know all of a sudden my compute fleet becomes more and more granular across the workload and i think any time that you've seen that in history is not sustainable and then all of a sudden right when you know as you look at that and now people are saying well as i go to reasoning models i want something that looks more like a cpu again so you know all of a sudden i didn't want that level you know that didn't become the predominant portion of my compute capacity i needed more of it over here right and model and model workflows and you know so on you know so i really don't like i'll say at scale specialization right because i think the workloads are going to continue to moderate you know and migrate so significantly so i think that's a good point and i think that's a good point and i think that's a good point and i think that's a good point first you know second is i think today's models are about to go through some evolutionary breakthroughs as well right you know i think the llm is sort of reaching the limits and now as people look to how do i really model 3d right you know where you know a flattish llm doesn't work well when you go to molecules and chemicals and you know right imaging you know so i think we'll see limits of that that'll cause different shifts in the algorithmic domain i also think that some of the most interesting workloads become where you start you know i'll say bringing hpc like things back into ai like things where all of a sudden you know 64-bit provision uh precision counts again right so i see what would be an example of that well you know imagine that my ai model is now inducing the five most interesting domains for a chemical analysis right okay those chemical algorithms are going to run with high precision right you know when you know which is going to be the dominant piece of the workload right getting to the five algorithms i need to run or running the five algorithms right on different chemical or biological systems so you know i do think that uh you know or as we optimize our ai systems they will lead us back to some of the things that have been more traditional high performance computing so i see that aspect of workload uh as well so you know all of that said is you know this extreme specialization to me in the compute architect i sort of don't like it because i don't think it's going to be what the workloads will look like two three four years from now so that's one reason i don't see all of these ai chips as they get more and more specialized for portions of the compute workload to necessarily be right you know second is right there's like a hundred of them now right you know there's multiple optical ones there's probabilistic ones and so on and you know they're not all going to win and they're not all going to win because at the end of the day you have to get scale yeah on these things and scale requires you know you have to win you have to get capital you have to get workloads onto it so i think it sort of defies logic that you're going to have a hundred of these things so i see them you know coming back uh as a result of you know simple you know capital market share etc you know which are the winning teams you know winning designs winning architecture so i see the workload driving that you know i see the natural you know industry consolidations you know that will occur and then i also expect that the winners will pick some winners right you know where you know an open ai and nvidia and anthropic will say i like this one because it isn't just hardware it's how do the hardware and software code evolve yeah right you know for it and i do expect that there's going to be hmm you know here's this 10 i could pick for this i like that one and it does require investment in the software and the workload evolution to take advantage of those platforms so for those three reasons yeah we're going to see you know a narrowing of the field uh and that's that's on
Speaker 3the deployment complexity
Speaker 2as well yeah sorry go ahead i mean it makes total sense and i think i agree in most part let me still try to stir a little bit controversy here just to to make it you know story controversy it's about no totally not me but you guys have changed it right yeah so so obviously a hundred companies more accessible consolidate right and and i completely agree with your argument that at the end of the day the large consumers will pick winners to some some degree right today i think you're right i think it's really important to be a first-tier chip startup to some degree whether you're you're that or not is defined by how much traction you have with the with the really big guys that's it um and there's also an argument to say like look in any kind of transition period, you probably have a bit more of an explosion of different approaches, and over time, that'll shake out. Now, all that said, I think there's an argument you can make that basically says, look, historically, we standardized on one or two architectures because different instruction sets were hard, you had to build multiple compilers. Basically, the software complexity dictated that you can only have a certain small number of hardware platforms. That part seems to be changing with AI a little bit. Today, writing an optimized kernel for, I mean, I think was a great example, but they basically said, look, humans can't really program this thing anymore, but an agent swarm overnight can do it. That's all we need, right? Where basically you can say, if you can build a chip that has a certain capability, it's now much, much easier to build the software layers and talk that fully leverages that. Could a result of that be that we have a little bit more heterogeneity? Just because it's easy to program that heterogeneity going forward.
Speaker 1I do think it changes where the bottleneck lands. It changes in that regard. You know, the first bottleneck is the one I'd point to that we've already talked about. Still takes a year and a half to build these things, right? And then the second bottleneck is not only does it take a year and a half to get these things into racks and scales and so on like that. Now, to make them useful, okay, well, could I borrow 50 billion? Right? I mean, you're not talking about small capital expenditures. You're talking about very large capital expenditures and data centers, you know, power, you know, commitments and so on like that. Well, I got to build this thing. That's scale, right? Associated with it. So, and that's going to defy a lot of heterogeneity as a result. Now, you know, clearly big players, and we saw this with what, you know, NVIDIA did with Grok. Okay, you know, we're going to sort of own some of that heterogeneity, right? And, you know, two years from now, you're not going to tell what Grok is inside of NVIDIA, right? It's just going to be like another special, right? You know, something like that. So, you'll see the architectures. You'll sort of hide the heterogeneity, right, inside of it, which to me is just fine, right? You know, I mean, that's what layering, that's what abstraction has always done associated with it. You know, but I think those forces are powerful ones to, I'll say, expose a lot of the heterogeneity? No, right? I think it will get abstracted underneath some of these. But at the end of the day, it's still hard to build these things at scale.
Speaker 3Earlier, you talked about HPM being hideous memory.
Speaker 1Mm-hmm. By far the best one we have today.
Speaker 3Exactly. The thing about democracy. We can't recall when there was the last memory innovation. It might have been opt-in. I think you probably had something to do with it.
Speaker 1Yeah, that one died.
Speaker 3That one died.
Speaker 1We killed it. Yeah, we killed it.
Speaker 3So, do you think memory innovation is around the corner? Or what needs to happen for obviously we understand the scale problem because it's even more larger there. But if you keep the scale aside, even from a technology point of view.
Speaker 1Yeah, and let's you know, look at that history. I've probably personally been associated with at least five different new memory architectures, opt-in just being one of them, right? You know, that did not see the light of day, right, as well. I'm probably familiar with close to a hundred that have happened in the industry over the last 30 years. And exactly how many major new memories have we had over the last 30 years? Zero. Right. Which, you know, okay, DRAM, SRAM, FLASH. Okay. What else is there? DRAM, SRAM, FLASH. You know, so, I do think.
Speaker 3Flash stacking comes from a little bit.
Speaker 1Barely. Right, you know, but anyway, right. So, you know, in that it has been right, you know, a disappointing that we haven't created you know, the next physics that enable, you know, low cost, high performance, resilient, manufacturable memory, right, you know, for it. So, I do think and now and over the last 30, 40 years, you know, memory was a terrible industry. Right, because you would make money one year and then you'd lose money for the next four. Right, so, let's go do R&D in an industry that makes money one out of five years. Right, you know, it's just always, you know, it's just been so terribly difficult because it's been such, you know, driven by such violent commoditization
Speaker 2cycles. That has changed a little bit though recently. Over the last five years? Happy days, man. So, all three big memory vendors are on the top 20 most valuable companies on the planet. Yeah. That's a crazy change.
Speaker 1It's just so, right, you know, who would have thought, right, associated with it. And anybody who looks at the AI workloads and it's always, you know, it's much harder to, you know, it's easy to predict the past but it's a little bit harder to predict the future. Right, and, you know, in that, you look at it and say, AI is a memory workload. Exactly. Right, you know, inside of it. So, I think there is the combination of capital and technology needs that will justify memory innovation. Now, there's two domains that I'm excited about, you know, for, you know, one is, you know, this idea, you know, I do think that the shoreline bandwidth limitations of HBM is a fundamental issue. Yeah. Right. So, with that, the idea, you know, and obviously, you know, we have a company, D-Matrix, that's doing this, but I think others will be, you know, whether it's, you know, Cerebrus, you know, and their techniques. The future is stacked. Yeah. Somehow, you got to bring memory into. In more ways than one. Yeah. You know, you got to bring memory and compute together. Yeah. Right. You know, in fundamental new structures. I do think that given the workload now has such capital being deployed against it, you know, in the AI space, that I do think we'll actually be able to break through some of those physics challenges as well. So, I do believe, and, you know, I've just funded. You know, I've just funded a new memory company as well that's still in stealth, but, you know, I'm quite excited about some of the innovations that will occur in new memories, and you're going to see new materials, you know, for memory, you know, a lot of people looking at things like Ferroelectrics, you know, finding non-capacitive, you know, high-density memories, finding memories that can, you know, be high-performance, high-density and stackable, right? But, you know, people are looking at low-cost Flash, you know, for that. I'm not particularly enthusiastic about that. That idea, myself, I think, you know, speed and performance of the underlying cell are somewhat problematic with a Flash or an MRAM structure, you know, but I do think there's memory innovations are coming for the first time in 30 years, and I do think that's going to be an exciting space, and, you know, we'll have a couple of companies there, I'm sure. Yeah, hopefully. Yeah, hopefully. What do you think? Should we do it? Okay, you know, but we'll find some things to do here, but I think memory innovation for the first time in 30 years is nigh upon us.
Speaker 2Yeah, yeah. It's amazing what a couple of $10 billion of market cap, do we get $100 billion? A trillion, okay, sorry.
Speaker 1You know, I think the memory industry has now increased by market cap by $2.5 trillion over the last four years. I think that's about right.
Speaker 3That is, yeah.
Speaker 2Of course, just invested there and not done any of our venture stuff.
Speaker 3Yeah, exactly. How long, what's your view?
Speaker 2Sorry, go ahead. To expand a little bit on the stacking part, if you have to predict how quickly will our chips get taller? Is this going to be like, you know, we're going to go to two and then sit there for a while, or will we have, you know, the 16-layer sandwich, you know, happily mixing logic, memory,
Speaker 3and SRAM? HPM pulled out this.
Speaker 2HPM is going down at Moffat.
Speaker 1But how do you see this evolve? Yeah, you know, I think there's you know, the problem with stacking, right, is now, essentially, the value of the stack, right, becomes greater as you go to higher dimensionality of the stack, which means the yield of any individual layer in the stacking process has to keep coming, you know, essentially, it can't go up linearly, right? You know, it has to go up exponentially. And improving the manufacturing and defect characteristics in an exponential way, you know, when you get to eight or 16 high stacks, gets to be so, so hard when you run the math on that associated way. You know, your manufacturing flows have to be so close to perfect.
Speaker 2Although you could see designs that take into account a certain amount of cost, like we've seen it for a long time for two-dimensional.
Speaker 1Yeah, of course. But remember, that was the nature of DRAM. You always have bad block replacement and, you know, spare lines and so on like that. So not in any way, right, you know, minimizing that. But assuming you're doing that, you still need the yields to be so good. You know, if I have a cracked die, I don't care how much resilience you've built into it. You can't, you know, it's a bad die. Right. So, you know, those, you know, so as a result of that, I'm not, you know, a crazy 1632 stack, you know, kind of guy. I think there's going to be, you know, nice three, four or five stacks, you know, that just end up sort of being the sweet spot. The mid rises. Yeah, right. You know, for and I expect for part of its manufacturing, part of its complexity associated with it, you know, where's those yields, sweet spots associated. So that's sort of where I think most things land. But even there, right, you know, so let's say I now have, you know, I now have a great compute tile. Right. You know, how big does that tile need to be? I mean, everything's become radical limited now. Right. You know, can I go to more chiplets that better yield characteristics, better thermal, you know, and then you're going to say, I want a memory stack on that. You know, is it a two stack memory, a four stack memory? You know, I don't see it becoming an eight stack memory. You know, so to me, two or four sort of right. You But then power delivery needs to get integrated into that so that becomes sort of another element and i want that in the z dimension right associated with it because you know i want my power rails coming into this as well right and then i need you know fairly intelligent rdl layers to distribute signals across that platform so to some degree you know my canonical thing is already about eight layers thick when you add up all of those pieces associated with it and i think that's about all you can do right at least as we understand manufacturing today and then of course you expect optics to come in so you're going to have more 3-5 material structures that are going to be closer to this complex because i want you know my io getting you know essentially you know deeply integrated into that construct as well you know so my silicon centric pieces need to get complemented by some of these other 3-5 materials and how do i get optical connectivity built into this how do i get power rails you know built into that you know that is a you know i'll say that is an engineering feat and a manufacturing nightmare you need to bring all those pieces but that's where the physics is taking us uh over time so because of that you know throwing a 16 high stack into you know it's like that's crazy you know a two or four high memory stack you know i think that's probably where the speed
Speaker 3sweet spot will be and didn't talk about the thermal dissipation and new materials needed for that yeah yeah and obviously
Speaker 1there's going to be a lot of you know you know first thing right you just can't have some of the you know the physics hot spots that you have today so you have to be more intelligent about how you use this the heat and spread the heat but you know then you're going to be saying okay how do i both get power in and dissipate power out of those environments and by the way if i can you know you know get much more efficient power in at the voltages i want with much better you know power responsiveness of my voltage regulation you know then i don't create as much thermal chimneys yeah but i do have to think okay how do i cool these things and you know that's something that's going to be a big challenge for me and i'm going to have to think about how do i cool these things and you know that's something that i'm going to have to think about how do i cool these things and you know that's something that i'm going to have to think about how do i cool you know what role do new materials diamonds and other things have been postulated as ways to you know do a better job of the thermal cooling how do i then have better you know liquid cooling techniques and others you know sitting right on it you know with good thermal uh characteristics so yeah you know uh you know we we engineers are becoming plumbers right
Speaker 3exactly yeah you don't worry about mc the chip scale yeah let me ask you about an operate phenomenon which is some people have said hey let's go ahead and do this let's go ahead and do this let's go ahead and do this let's get the memory off of the package and just go all optical between the compute and the memory and that'll give you more degrees of freedom and more ways of building things what's your view on that as opposed to in stacking that's other extreme
Speaker 1yeah you know i think the more oeo right uh interfaces that you have you know oh you know optical to electrical to optical tool you know the more of those that you demand in your core compute complex enemy is just hard right uh in that sense and you're always going to have losses associated with it you know people are you know working hard to come up with more clever techniques to decrease you know both the cost and the power right you know how how much of the power am i wasting as i go through my you know my dax and uh you know conversions so i'm not a fan of those you know because i think it gives you access to large pools of memory right but i burn a lot of power getting there yeah right and remember my you know the essentially you know the uh femtojoules per compute is now actually pretty amazingly good right the femtojoules per bit of communication is comparatively a thousand times worse right and if i'm spending that much to go get to large pools so i don't like those approaches you know myself i think they somewhat defy the underlying physics of the solution i mean and
Speaker 2then there's sort of the other approach to say like look can we actually integrate compute and memory directly right shift some of the compute into the memory it's some of the pin approaches yeah exactly right so what what are your thoughts about that i'm
Speaker 1not a fan of those either right um because i think workloads you know because now i have to take my workload and i have to like push these little pieces of compute into the memory right you know so that you know i don't need as much bandwidth and memory coming into my real compute right and that puts constraints on workloads and again you know i could be wrong on this particularly some of the design flexibility and the intelligence that we can now bring you know through the the design aspects but most pym ideas you know have been around for 25 30 years and i think 25 30 years from now they'll still be hanging around right in that sense you know give me great memory bandwidth you know that i can have you know high density uh high performance and get it close to the real compute engines i expect that's going to be a more generalizable solution across many and evolving workloads over time so we're
Speaker 2still stuck with we're still stuck with you know we're still stuck with you know we're still stuck with you know we're still stuck with you know we're still stuck with you know we're still stuck with you know denser and denser chips more compute more logic
Speaker 1more power and but much better io now i am a i am a huge fan right of optic you know moving optics for all io function yeah right you know i declared the death of copper about 25 years ago eventually i'll be right you're still right yeah yeah one day soon uh i'll be right but i think is that it you know to me all i o should go to optical right you know but that core compute memory complex no right
Speaker 3so when do you the first step for going um on optical is to scale up even more so than scale out yeah yeah what's your views on that yeah you know and i
Speaker 1think we're now what would be
Speaker 3the like the application of the workload driver if you will well i
Speaker 1think the workload driver is the one in front of us okay right i mean you know today you know our scale up environments we're putting tons of copper right you know and essentially copper is becoming a wave guide that's getting shorter and shorter right and all of a sudden you know to make my copper work at uh you know five meters is more expensive than making optical work right at 100 meters right you know so you know i think the physics is driving us there today you know and you know so if that's the case pat why haven't we moved there already yeah it's a complex supply chain that you know has never been tested at scale and i think now you know i think everybody is looking at you know some form of npc o cpo etc you know in the 28 29 time frame because you know i can't scale up to large radix you know compute uh clusters from my scale up environment without making that move so you know clearly nvidia has uh indicated that i think everybody else in the industry expects you know that's the conversion point you know in reality i should have never built an nvl 72. you know it's an engineering marvel and it's a manufacturing nightmare right you know it took the industry you know one extra year and a half to digest that beast right it really is amazing that you know we're able to get it to scale and so on but you know that really proved that had we had a more mature optical supply chain we'd be much more advanced today we'd be able to build better scale up environments today you know better you know power better cost better energy but we didn't have the maturity of the optical supply chains to allow that those things are not trivial to get resolved to happen but i think 28 29
Speaker 3is the capacity
Speaker 2we're currently gaining a lot of experience how impact optics work and networking right and and i think many of that will translate nicely yeah
Speaker 1yeah i do think but all of a sudden you know it's like okay these things have to work you know at 10 million gpu scale wow right you know it has to fit in packages and you know there's just a lot of things there you know and many forms of optical don't like bad thermals well guess what bad thermals is the nature of the game for it you know so we have thermal problems to deal with you know package integration problems we didn't have enough you know laser capacity in the industry so a bunch of things but i don't see anything there fundamental it's just a lot of hard work at this point to pull all of those pieces together in terms of
Speaker 2the networked architecture will this also mean a switch what's more circuit switch networks just because you know you you can't scale these these very fast optical networks indefinitely
Speaker 1well you uh you know i i like uh you know uh nick mccune you know another stand for professor here a good friend of mine you know as well you know i mean you know him a little bit also uh agreed on you know so but you know to me he describes it well you know we built you know uh networks for to be able to handle any packet going anywhere with no knowledge of where it might go when you think about ai you know it's almost exactly the opposite right you know we're doing flows that are large and predictable so in that sense just the very description says hmm that looks more like a switch than like a packet so i do think this confluence of moving to optical right and you know having a workload that is far more predictable manageable you know large and flow you know to me you know i do see that yeah we will go all the way to bringing optical into the compute complex and we will go all the way to ocs or some form of optical uh switching you know with more predictable flows you know to me that is the right you know architecture and then you sort of say is that scale up or scale out and to some degree it sort of dissolves the boundary yeah right if my scale up is big enough then yeah you know just saying okay what's the radix in my scale up environment and when do i want to hook it up to other clusters that are really big and to me that ends up being a little bit more complicated than what we're talking about being a pretty beautiful answer uh you know to the future but i'm looking at somebody who's way more expert on this than myself so
Speaker 2No, look, I always found scale-up versus scale-up to some degree an artificial distinction. And you understand where it comes from, right? We're kind of using different protocols. One is front-end, one is back-end network and so on. So right now, practically speaking, it is a big difference. If you take a step back and look at it from a system architecture perspective, it achieves the same goal, you know, with slightly different flavors, slightly different protocols that over time I think will converge.
Speaker 3Yeah. I mean, especially for the AI training workload.
Speaker 2Yeah, although, I mean, training and inference actually look fairly similar. The inference corpus is less similar.
Speaker 1Yeah, now I'm going to, you know, right, does your brain stop learning? Right, you know, by the way, it's another characteristic of workload evolution. I do think the, you know, the separation of training and inferencing is going to become, you know, less so going forward, not more so, right? That's a little bit of the argument against, you know, more and more heterogeneous architectures is I think continuous learning. You know, I think there'll be algorithmic domains that will say, yes, continuous learning, you know, becomes the nature of the algorithm where, yeah, I do want my inference environment to be updating my model weights because, you know, that's giving me more and more refinement, you know, evolution across, you know, workloads, moving into environments that are less formalized training as well, we'll have smaller data sets in those environments. So, you know, I'm not sure that's a.
Speaker 3But the number of nodes you need in order to be useful is a smaller. diameter for inference. That's the training, that's what it means.
Speaker 2Today, that's true, yeah. Yeah. I mean, I still think there might be a role for heterogeneity coming from a different angle, which is that different types of AI workloads require a very different underlying world. Like some are more compute limited, some are more memory bandwidth. We have a small diffusion workload also, right? The optimal card for that looks very different from, you know, a top of the line, you know, $15 trillion, sorry, $15 trillion weight LLM, right? That means. So it's a great massive scale, right? So it's. I think that might drive heterogeneity, if anything.
Speaker 1Yeah, I think there's some room for that. Yeah, but I'd also say, okay, now I'm talking about putting a million GPUs to work in my data center, right? And you say, well, you know, for that particular diffusion workload, you know, I should really put something optimized this way, right? So I'm now going to start, you know, putting, you know, a thousand of those in, right? And for that workload, I need, you know, 2,000 of these in. And, you know, for some. You know, and all of a sudden, you know, you don't get some of the, you know, the flexibility of scale, right, that you would say. And, you know, how much does it cost me to manage and upgrade and, you know, connect and, you know, fault domain, you know, all those types of things. The agents will do all the upgrading, so there's no longer a problem. Yeah, yeah, yeah. Thank you. Everything gets easy. So good, I don't care.
Speaker 3Yeah, so fleet management is done. So we talked about what? Compute, memory, networking, power is the next one. Yeah, yeah. And you touched upon it with vertical delivery inside the chip, on the chip. But, I mean, there's 800-volt DC, there's power delivery to the data center. There's an entire ecosystem all the way to power generation and the sociopolitical dimensions of that and so on and so forth. Yeah, yeah. So where does innovation come in and where does, like, it's pure execution?
Speaker 1Well, the, you know, and if we start at, you know, first principles here, you know, our nation has done a terrible job with its energy capacity, right? You know, and essentially we went through 10 to 15 years where essentially I was taking coal offline at the rate I was adding renewables and essentially the nation was flatlined in terms of overall energy capacity. Not good. Because in a AI digital age, energy capacity, equals economic capacity, right? So essentially my economic capacity as a nation, flat for 15 years if you buy that thesis, right? Which I think is very, you know, provable. Over the last five years, I've, you know, we've seen this enormous influx into just give me more capacity. But, you know, I think, you know, even with that, we've gone to maybe increasing our national energy capacity 4% per year.
Speaker 3Something like that.
Speaker 1Going from, you know, essentially zero to one, right? Wow, 4X or 4%, right? So to me, you know, that, you know, it's a bad situation. So we just need more energy capacity for it. So number one, and obviously we have some companies in that space, you know, we talked about nuclear upgrading as one of those with Alva. And, you know, I think nuclear, right, you know, as a baseload is, you know, a fabulous thing that we want to go build more of. You know, unfortunately, all of our renewables have, you know, deep dependencies in China, which is very unfortunate, you know, for us, you know, where, you know, gas turbine lead times are only eight years at this point, right? So, you know, I mean, we really have a difficult environment to scale up rapidly. But one, we just need more capacity. And fundamentally, energy capacity somewhat dampens how euphoric we can get on AI, right? Because, you know, why build a new data center and buy, you know, the million GPUs if I can't power it? I can't power them, right? And I think you're going to see more and more defaults happening on many of those data center projects because the energy won't be there.
Speaker 3So you think it'll be a significant headwind?
Speaker 1I think it will be a headwind. And I think you're already starting to see some of the first indicators of that, the Oracle, you know, comment, you know, which is, I think that's the first of what you're going to see many. Now, it's not like we're going to slow down as a result, but I think people will start saying, do I really have the energy, you know, to underwrite the capital commitments that I'm making? You know, for, you know, cement, for data center racks and GPU purchases. So I do think it's, you know, going to be, you know, an upper bound. But we need more energy capacity and we need to get more innovation into that and we need more of those supply chains in the U.S. You know, nuclear, one of my favorites. When was the last nuclear reactor that came online in the U.S.? Right? You know, it was 20 years ago. Right? You know, we just stopped, you know, a very capable industry. So that's one we need more. You know, we need to get much more efficient at delivering, you know, the power networks. And as you said, you know, moving to 800-volt DC data centers. Absolutely, you know, right? You know, and being able to eliminate so many conversion steps along the way because of historical, you know, reasons and, you know, standardization reasons. Okay, we got to fix this. You know, an 800-volt DC, the revenge of Edison is upon us. Yes, I was just going to say. Yeah, I like to say. So, you know, I do think that ends up being, we need better power conversion. You know, we have a couple of companies, one of those vertical, you know, vertical GAN, you know, I think will be, you know, a killer technology, you know, to be able to do 800 to 48 or 800 to 12 or even 800 to 5 in one conversion step. Right? Which just gives you a lot more energy. You know, we touched on. Solid-state transformers. Yeah. You know, all of those are, you know, solid-state transformers, switches, et cetera, you know, rebuilding, you know, the power delivery network. And I think that will, you know, have a lot of, you know, innovation, you know, sort of like, okay, power's cool again, right? You know, forward. And then you're going to need all of those in the cooling systems as well, right? You know, and so there'll be a bunch of innovations in that, you know, so, you know, you know, turbines and refrigerants and, you know, all of a sudden they get to be, you know, exciting new technologies again. So, you know, for all of us hardware guys, this, you know, this, you know, it's like renaissance in Broadway. It's going to be, you know.
Speaker 3For 25 years, I had to go back and read about Cardinals.
Speaker 1Cardinal efficiencies. Oh, yeah, this is like good stuff. How does that first law of thermodynamics work again? The physics of that might be worth something after all. You know, so it's going to be, you know, an exciting time that way. You know, but, you know, we also say, you know, we need to change the physics, right? Because CMOS hasn't improved, you know, the essential power per tera, you know, power per tera flop for the last five generations of NVIDIA chips, you know, flatline. Not okay, right? And that's where some of the things, you know, we have a superconducting company, Snowcat, right, which affords the opportunity to fundamentally change the physics, you know, a thousand times better power performance. So I do think some of those innovations are upon us in the near future as well.
Speaker 3So I've been a terrible time manager here for the past hour. So, so many topics yet to cover, we didn't cover, but we got to cover at least one before we leave. Okay. Because now.
Speaker 1You're in charge.
Speaker 3Yes, so I thought. Now VMs are back. How do you think about that? As a former CEO of VMware.
Speaker 2Well, you know, I. I feel excluded from this conversation. No, no, no, I was just. Former CEO of VMware, if we go on.
Speaker 3Part of it was I was going to talk about networking as well. So, but I don't think we have the time.
Speaker 1But, you know, hey, you know, I think the, you know, every innovation has always led to the next abstraction, right? And I think many of these ideas and, you know, right, you know, I would like to say, you know, you know, what's new? What's new in our world today? You know, data structures, algorithms, you know, an abstraction, you know, you know, we bring it back to those, you know, foundationals. And, hey, you know, I think, you know, the virtual machine abstraction, you know, whether it's done from the infrastructure level or from the application level, you know, I think it's still a foundational abstraction model, you know, that deserves to always have a place in the compute hierarchy. So, you know, so what do you think? One of us go back and run VMware?
Speaker 3Yeah, that's what I tell, I think I told Guto's home, to go back to what we ever had. -
Speaker 1Yeah, yeah, and you know, And I do think some of these abstractions, you know, but you also think about it. And like one of the things we did at VMware, we, you know, essentially, you know, managed every aspect of the computing hierarchy. Right. You know, and you manage the workload, you manage the network, you manage the storage system. And how do you abstract those? And I think in the AI context, right, we think about, you know, these agent swarms. Well, who's going to manage the agents? Who's going to create the security profiles around all of the agents? Who's going to manage the performance of the agents? You know, essentially, every fundamental element of virtualization and management needs to get recreated in this next computing hierarchy. And there's going to be lots of wonderful companies that, you know, sort of break through for how those things will be, you know, done.
Speaker 3Okay. Do you want to end it on networking?
Speaker 2Actually, I'd love to stay with the VMs for a second. The most interesting thing for me is that we're now building VMs so that they're usable for agents, as opposed to usables. And what we've seen in sort of other companies is that the changes a lot, the form factor that changes a lot, how you market them, right? You want the agent to make the pick for your VM offering. It changes a lot how much complexity you can have. It changes how quickly the startup should be. Humans are a lot more patient than agents. So it's, if you have to rebuild VMware today, how would it look different? Yeah. And I do think. VMware for agents, right?
Speaker 1Yeah. And I do think there's. Does there need to be a VMware for agents for humans, right? I see what you mean, yes. Right. You know, in the sense that, you know, I need humans. Let's stick to the agents. They're way more interesting than the VMs right now. Yeah. But, you know, I think it's an important point because I do think that there needs to be, you know, and I'll use, you know, Dario's constitution, right, or guardrails, right, or being able to set policies. And I think then you really say, okay, every aspect of the virtual machine, as we know, we love it today, it is servicing agents, right, as opposed to servicing hardware and humans, right? So I think, you know, that becomes your fundamental design constraint on this side. You know, how do I make agents great, right? How can I make them secure? How can I make them performant? You know, how do I abstract them? How can I migrate them? All of those things, you know, have the new embodiments, you know, vMotion for agents, right? It's like every one of these will have the new embodiment here for them. But then I also need to think about how can humans be setting the policies, describing the constitutions, you know, getting the, you know, dashboards, et cetera. And I think that's sort of, you know, where the interesting thing is because you need to do both of those, right? Not just one of those, but the fundamental, you know, I'll say the hard design stuff like we did for, you know, virtual machines was associated, abstracting the hardware in a performant way for applications or operating systems. Here it's, you know, abstracting the hardware and the operations for agents. Yes. Fundamental thing that you're serving.
Speaker 3It was great discussing this and thank you so much, Pat, for coming in and we'll do it again.
Speaker 1Well, two of my favorite people, anytime. All right. That was awesome. Thank you. Thank you.
Speaker 4Thanks for listening to this episode of the A16Z Podcast. If you liked this episode, be sure to like, comment, subscribe, leave us a rating or review, and share it with your friends and family. For more episodes, go to YouTube, Apple Podcasts, and Spotify. Follow us on X at A16ZPodcast. And subscribe to our Substack at A16Z.Substack.com. Thanks again for listening. I'll see you in the next episode. As a reminder, the content here is for informational purposes only. It should not be taken as legal business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our A16Z podcast, please visit our website at www.a16zpodcast.com. For more information on our investments, please see A16Z.com forward slash disclosures.
PodcastTranscript AI