Data Centers: The Hidden Backbone of Our Modern World

All right, and I, well, kids are in school. Kids are in school. We survived the summer. I love how back to school kind of feels like the beginning of the new year. It does. I always thought this time of year is the actual new year. I like that. (upbeat music) Welcome to the third episode of The Stepchains Show. We're here to cover the stories of human progress. We want to understand the technologies, systems, and infrastructure that shape our world. And I'm Ben Eitelson. I'm a co-founder of Stepchains Ventures, a fun and invest in the companies that are accelerating today's biggest stepchangers. And I'm based up in Seattle, Washington. - And I'm Anai Shah, fellow co-founder of Stepchains Ventures and based in Los Angeles, California. Today, we're gonna be telling the story of the hyper-scale of a recent infrastructure. The story of data centers. Every time you stream a movie, send a text message, call a car to pick you up or talk to what feels like a fully formed computational consciousness. You are touching an invisible physical empire. We call it the cloud, but it isn't in the sky. It lives somewhere very, very real. - It lives in nearly 12,000 buildings worldwide, consuming almost 5% of electricity in the US. And it lives inside these garden hose size cables that are laid across the darkest parts of the ocean floor. Today, we are going to tell the story of this invisible infrastructure. A story that begins in the humming and clattering punch card rooms of the early 1900s, winds through the Cold War projects that accidentally birth the internet and leads the gigawatt scale AI factories, the size of lower Manhattan that are being built today. - The US stock market is worth around $60 trillion, made up of over 4,000 companies. But there's just six companies at the top, Nvidia, Microsoft, Apple, Alphabet, Amazon, and Meta that make up 30% of that market. They are today's railroads, steel, and oil companies and are building the modern industrial engine of our time. - This is the story of Data Centers. But before we do, just one quick note. We have all of the research links and notes for this episode at @stepchanged.show. And if you're listening to this and you think of a friend or colleague who might enjoy it, please send it over there away. All of this is a new endeavor for us, and we appreciate it getting into the hands or rather ears of folks who may dig it. All right, so Data Centers. - So Ben, I was recently reading an article saying these new technologies are saving work for everyone nowadays. Pretty soon we'll have nothing to do at all. That's right, I think they were saying that we have some new electronic brain that's gonna make most workers obsolete. - Right, but the CEO behind one of these technologies quickly countered saying that it was a way to save time, not replace jobs. He said it was a small tool to help great minds benefit mankind. - Was this the CEO of Anthropic or Open AI or maybe Google? - You might think so, but no. This was Thomas J. Watson Sr., the original leader of IBM. - So IBM, the computational powerhouse of the last century long before we had the Googles and the Amazon's and the Metas and Microsoft's, there was big blue IBM. And long before IBM ever built an electronic computer, they were doing computation of a different sort. And if you walked into a large company back in the 1930s, you might find yourself in one of their windowless rooms, humming and clattering like a small factory. They'd be long rows of metal cabinets, worrying gears and clerks, feeding stacks of stiff paper cards into these behemoth machines. Each card was a sliver of information, an employee's hours, an invoice, maybe a customer's address, but together they formed the first centralized nerve center for corporate data. - And I think that's why we argue that those rooms with those cards of information being processed on these devices that we're doing computing are the first real data centers. That machine has quite an interesting history born to solve a government problem, which is the problem of the US census. - It's wild, I mean, the ETA census, they would collect all the information in 1980. And it took seven years before they got it tabulated. And so you realized it just was not working at human computation scale anymore. - And so the one man, Herman Hallrath, had an invention to solve this. - He took the way that railroad conductors checked tickets and noticed that these were papers that had punched out holes that ultimately stored and represented information. And if you could build a machine that could count the number of punch holes, you could then do computation at scale without needing to have humans count mark downs on a form. - That's right, these cards stored information, and then you could have machines compute off of them. And then 1890 census was the machine's first big break. Even with a population 25% larger than the decade before, it was completed in two years instead of more than seven years. And it came five million dollars under budget, which is not something that typically a new technology could achieve for a government. And so fast forward to Watson, he sees the punch card machine and he understands what it can do. And by the mid-1920s, he's convinced. - So by 1924, he we branded the company international business machines, a much more fitting name than CTR, computing, tabulating, recording company. (laughing) - There is no limit for this tabulating business. He told his executives in 1927. And so he doubled down. He sells off all the other less promising lines of business and pours the resources of this company that he's running into the tabulated division. They design a proprietary 80 column IBM card that only works on IBM machines to become the IBM card, a sort of early days lock-in, where customers who wanted to use the machines had to buy the cards. Everyone obsesses over the Gillette, razor blade business model, but not only is the IBM card how you do the computation, but it's also storing information and it becomes your record-keeping method inside of businesses that are running on these IBM machines. Once you put invoices and time cards and all of that information onto the IBM card, you're not migrating it off. It's on a piece of paper. Yeah, that's right. So now you've produced a very steady stream of revenue alongside selling the tabulator machine. And the timing for this was perfect. This is the late 20s, early 30s. The new deal brings in massive government focus around record-keeping. And in 1935, the Social Security Administration signed a contract with IBM requiring millions of their cards and machines to process benefits and print checks. One of their New York plans is soon printing 10 million cards per day. Amazing. And within a couple decades, most large companies had punch card rooms. They had various machines they used to sort, tabulate, and store financial information, payroll information, employee information, customer information, everything you think of. And then in order to manage that, they'd hire clerks and technical specialists to operate and maintain it. It's easy to take for granted. But this was the first time a company and it's accounting and it's invoicing was being able to be calculated at this scale. And you think about what's going on in this era. And this is when companies are starting to scale in new ways. It's also when more global trade is starting to happen. Thomas Watson's senior ends up very interested in diplomacy and does a bunch of traveling around the world trying to preach that commerce across borders is going to create peace. Unfortunately, this backfires need a lot of commerce with Germany and the lead up to the Holocaust. And there's a lot of different stories around IBM's computational power being used to help Germans with record keeping. And ultimately, this product is foundational to so much of the scale of this era, including the scale of war. Every time there was a war or a big government project, regardless of which country it was fighting, they needed IBM machines to do calculations. This was the way to process and store information. But it was all still cards and mechanical switches, ultimately. And rooms full of paper. And rooms of paper. So everyone's running on punch cards. What is this thing? What is this room? What should we call it? This is where we're going to put our stake in the ground and say the earliest data centers come out of the IBM punch card. A data center, the easiest way to understand it is it's a physical space or a collection of facilities designed to house and operate an organization's data in computing infrastructure. And so you can think of it as primarily storing information, computing information, and over time connecting and allowing for communication of information. At its basic core, a data center in the evolution that we're going to talk about from the mid 1900s to present day is the physical space around storage, computing, and connectivity. At this moment, IBM was already this growing company of import, right, doing global international business. By 1945, they had around 25,000 employees and annual revenue of approximately 140 million. But something was about to change. And that's also coming out of the war. World War II was this big moment of government investment and R&D. One of the things that they found themselves often doing calculating artillery firing. tables, which take into account the wind speed and the weather and all of these different factors to figure out where they should shoot artillery. And so there'd be teams of human computers, clerks that were doing mechanical calculation to figure things out. But that was a bottleneck. And so the Army funded a team at the University of Pennsylvania to build the first real electronic computer, a machine that could crank through those calculations in order of magnitude faster. And that became the electronic numerical, integrator, and computer, also known as the eniac. This was all about math. It was focused on calculating numbers and producing mathematical results that will help them, in this case, be more efficient in war planning. Yeah, I mean, they didn't have the TI-83 or the games that go on it or the games that go on it. They didn't have Tetris. So they needed the calculator that could go faster than the people that were a bottleneck at this point. And so the eniac had no mechanical parts that slowed its operation. It took up nearly 2,000 square foot room, but it did operate over 1,000 times faster than any previous computational device. And so it could execute 5,000 additions per second. At this time, IBM, which had the commercial product, the quickest punch card machine, could complete just four additions a second. So it's not order-magnitude faster. It's three orders of magnitude faster. Four additions per second to 5,000 additions per second. This seems like something that mathematicians and CEOs that are privy would think, "Oh, I can see that future." But no, a distinguished Harvard mathematician dismissed the idea as foolishness that there would be a big market for computers. He believed that the country would need maybe half a dozen, mainly for military and scientific research. Our friend Thomas J. Watson Sr., he said, "General-purpose computers had nothing whatsoever to do with IBM, or IBM's main line of equipment and profitability." But Watson's stubbornness to stay wedded to the past was only match and maybe outdone by his sons, stubbornness, to push the company forward. Runs in the family, but takes a different shape. This is a father and son story for the ages. Thomas Watson Jr. eventually goes and fights in the war, comes back, and it is ready to take his seat as a leader inside of IBM. He views electronics as the future of the company, and this becomes the intergenerational battle for the control by IBM. Thomas Watson Jr. saw that there is a future for the company in electronics and wanted to push it that way. There's some interesting parent-child psychology where you can see him completely frustrated and angry that his father's not going to jump on what he thinks is very clearly the future. IBM did do a wartime project with Harvard mathematician Howard Eiken. It was a hybrid, electromechanical vacuum-to-machine. They followed that with their first machine that they really wanted to demonstrate to the public after the war. This was called the selective sequence electronic calculator, SSEC. This was in 1948. They showed that in this Madison Avenue showroom right in the middle of Manhattan so that people could walk by and see this thing computing. By seeing this thing, these machines are still the size of rooms. It was fast. Not by today's standards, but by those days it was very fast. It didn't have any memory in the sense that we were used to computers today, having memory, but it had this punched paper tape as a form of storage. Despite Thomas Watson's senior not liking this as the new business, he didn't mind the company getting some PR points for it. It sent reporters down to watch the machine do computation. This is the moment when the press said, "Pretty soon, no one's going to have a job to do." As a result of this electronic brain. The electronic brain that would displace workers. In 1951, IBM's longtime customer, the US Census Bureau, went with Univac instead of IBM tabulators for its neck census. Thomas Watson Jr. recalls this moment saying, "My God, Univac is smart enough to start taking all the civilian business away." This is really what shakes IBM to once again reinvent itself. You know, you go back 1880, 1890, this whole invention was for the census problem, and that problem gets harder. More people and more things you want to know about people. You want to track more factors. And so this is actually a perfect testbed for computation at this time. Here comes the Uniac team that now is commercialized with Univac and wakes IBM right up. This company's been growing, right? IBM in 1940 was doing about 45 million in revenue with 12,000 employees. And a decade later, they're doing 250 million revenue with 30,000 employees. And if you're betting the company on a new direction, things could go south for a lot of people. It's a big responsibility for Watson Jr. And they pushed forward. And so out of these early computer experiments, they finally built their first commercial product, which is the IBM 701 electronic data processing machine. This is the first real machine that, "Hey, we're going to build more than one of these, and we're going to sell some of them. We're actually going to try and make this into a business, not just a research project." And as we'll see with some of the companies closer to the present day, they had an installed base of customers. They had a sales machine. And so they started this business line. And within five years, they had 85% of the computer market. So it's a team that invented a first vacuum tube computer in the ENIAC and then the Univac. They got outrun by the better commercial go-to-market sales machine that if you remember IBM was founded with. So who was buying these things? And how many did they sell? Well, the 701 was still a pretty bespoke product that they sold 19. And these were going to national labs, the weather bureau, and a lot of aerospace firms that were doing a lot of calculation. And the whole business model was actually not around buying these machines. It was an extension of the tabulating business model that IBM had always had. But they want you to lease the machine. They want you to rent it and buy punch cards. We got hardware as a service. Exactly. It was a beautiful business model. The era of the 50s for IBM was going from this research-centered product to what I'd consider their Model T. So in this era, they built the first disk drive. They started using tape in addition to punch cards. They wrote for-tran, which is the first real kind of modern programming language where you could write words and it would get translated into computer instructions. By the end of the decade, in 1959, they launched the IBM 1401, which was available to thousands of companies and they ended up selling around 12,000 of those machines. They exited the decade with a real computing business. And they're dominating the market. So that first machine shipped 19 systems. They had another machine that shipped 123 after that. Then the first transistorized Model T, the 1401, crossed over 10,000 units. From 1950 to 1962, IBM's revenue rose tenfold from $260 million to $2.6 billion. Their head count went from 30,000 employees to almost 130,000 employees. Woo! And so what else is happening right now? We're in the 1950s. World War II has ended. The Cold War is getting colder. And we find that the Soviet Union's long-range nuclear-armed bombers are able to cross the Arctic and reach American cities in a matter of hours. So this makes the decision window to detect and intercept down to minutes. It seems to forget that the Cold War was primarily a technological arms race. Who's going to get there faster if something's going down? If the Soviet Union launches these bombers, how quickly can we respond and know about it and then respond? And our Air Force and our defense systems were not designed for that speed. Yeah. We forced technology upgrade, forced our government to respond to invest. We had a laboratory project at MIT that in 1951 could process live radar data in real time, proving that automation could close this gap that we were seeing in national security. The challenge was how do you take this research prototype and turn it into a 24/7 machine that can operate at government scale. And this is a moment that becomes the largest computing project to date. Thomas Watson Jr. saw this and he knew that they needed to win this contract. IBM actually had a policy of essentially making only 1% profit on any defense work that they kept throughout this. But they said this is going to push us to the future. And so the contract awarded in 1954 became one of the largest contracts ever worth more than $500 million in 1950s dollars, around $5.5 billion in today's dollars. And so how did it work? What did it do? It was a full deployment when they got there in the 1960s. It was a network that's been 27 different centers where each center had a pair of special IBM computers that were designed for this. In case one of them was being serviced or down, the other one would become primary. So they built redundancy into the network. And the scale was unprecedented. In total, it was 56 computers. These were acre sized floors, multi-megawatt power draw to these systems. And they were connected. They were connected over early modems. This was the moment that really drove the production of the modem, where you could actually have data sent over telephone lines. And so they would lease special lines between these centers. And these centers have characteristics that we're going to talk about more. You have redundancy built into this. You have building level scale. You've got multiple facilities networked together. These end up being a lot of the same characteristics that drive data center growth and evolution decades later. That's right. And I feel like there's the scale of it. But I think the connectivity of this system is the difference between you have stories. you have compute, which was happening. But now you had connection and communication. Because you want this center over here to respond to information that was computed over there. And so this moment in history, the Sage system, I think is the first real time that machines were communicating with each other. Pretty big moment. So once we're able to do this in government, this is going to catch the eyes of other industries. The airlines have seen the Sage project take off. And they're sitting there with a new problem on their hands. This is the 60s. Air flight is booming. And the way that reservations got processed was absolutely insane. You would call your travel agent. You'll travel agent and call the airline. And the airline would have to run clerks around pulling cards out to book a seat. Supposedly it took up to 90 minutes. Well, you got to figure out who had the middle seat in row 17. Who had the middle seat? Exactly. How are you going to figure that out without looking up the card? As American Airlines scaling to a real operational scale is starting to break down. So they reach out to IBM and they kick off this project that'd be known as Saber. It was essentially like a commercial version of the Sage system. Two IBM mainframes were purpose built would be connected over phone lines. Those systems would house the source of truth around the reservations. And there would be terminals. These weren't terminals with screens. These are terminals with paper where you'd still have a travel agent, but they would be querying over the terminal. What seats were available and things would be booked automatically. So it'd go from 90 minutes to seconds at a terminal. And it removed the clerk in the middle. This set the stage for e-commerce. This is the first time I think you're buying something over a computer. You had 90 minutes to do a reservation to now down to seconds. And this scaled American Airlines operations by the mid 60s to be able to do 40,000 reservations per day. And saying this system continues to this day to be part of the backbone of airline booking. It's quite an amazing foundational moment where commerce is happening between data centers. And so they're running airlines. They're doing these major government projects. Businesses are now buying the 1401 machine off the shelves. But there's a different problem you enter the mid 60s with, which is all of these mainframes had their own accessories. There was no compatibility. So in 1964, they launched the system 360. And that turned this chaotic, messy ad hoc world of different mainframe models into a platform. It was essentially one architecture. You could choose how powerful you wanted it to be. By your base package. And then you featured up based on what you need. That's right. And so for facility planners, they could now plan a room in a facility and then be able to scale up the machine as needed. I believe roughly $5 billion of investment in R&D you have spent in this era. Another bet the company moment. And it worked. They were shipping thousands per month of these devices in the 60s. So you zoom out, you look across the 50s and 60s. We talked about earlier from 50s to 62. They scaled a 2.6 billion. By 1970, they were doing seven and a half billion dollars in annual revenue and had a team of 270,000 employees. And today's dollars that 62 billion in revenue. And the market noticed IBM was king. At that moment in 1970, IBM accounted for 6.8% of the total US stock market. It is almost feels like an unprecedented thing to say that today and video is currently 7% of the US stock market. These mainframes, they became part of the zeitgeist. They became part of a cultural milieu because they were often nicknamed the glass houses. You'd have the mainframes and their operators in an enclosed room. And as we mentioned before, you'd have windows and glass walls around it because you wanted to show off how automated your systems were, how advanced your company was. There was a recognition that these glass houses, these mainframes were powering the billions of dollars of revenue from their enterprise customers. And so the security, the operational control, the climate control, cleanliness, it was all part of this cultural moment where you had mainframes and these early data centers become a meaningful part of how our society and economy ran. If any company was using it effectively, they had more demand for use than they had supply in the computer. And so oftentimes people would be sitting there waiting for their information to come back for their computation to run. Sometimes it would be a day or longer. So the utilization was pretty high, but there was this long interactive latency problem. And an invention that echoes through in multiple ways to this day changed the fundamental way that computing was thought of. From this single threaded, I'm only working on one problem out once to know the job is to make sure my hardware is as utilized as possible. I don't care as a machine whether or not it's the same problem from the same person. I just need to be calculating. I need to be operating all the time to utilize my capacity. And you can be for this problem, I can free that problem, you can slot them in and out. It doesn't matter. That's right. And so a team at MIT in the early 60s developed what was called CTSS, the compatible time sharing system. And turned it on to the campus in 1963. And so students could sit there with electric typewriters and interact with this machine multiple at the same time and feel like they had control over the computer. It felt like it was your own. And this was a huge moment. This is the first time there was logins and users and files and instant feedback. And so it was revelatory to be there in the mid 60s experiencing that after the idea of a computer, it was just this kind of operate thing by someone else. That's right. You had to queue up, get in line, hand over your data to the operators, the specialists who would then go and do the thing and you would sit and wait. Now you're interacting. It is you and the computer. And the way this worked was that you didn't obviously have monopoly control over the computer. The computer was switching what problem it was working on depending on what free cycles it had. The human perception is like, oh, how can a computer do that? Well, the computer is operating at a much faster frequency than where you're able to realize. Even if it's something as simple as responding to a keystroke, there's plenty of milliseconds in between my keystroke and your keystroke for the computer to respond and then switch back to the other problem. And as we'll see later in this story, this notion of intelligently slicing up the hardware and utilizing it becomes the backbone of everything and how the cloud and data centers are architected today. And it started with this time sharing innovation. Well, we had these main frames that have taken over the business world and time sharing. So multiple people can be on a campus connected to that main frame feeling like they have their own computer. They were still fully disconnected from anyone else in the world. There were still islands onto themselves. So yes, there's now now on this island, multiple people can be there, but it's not like you can go visit the next door island. What are we missing? It's something we take quite for granted. Many people in different places, but working on the same system at the same time. Now remember, we are in the late 1950s in the Cold War and Eisenhower wants to ensure that the US would not be blindsided by a technological surprise. There's this particular moment when everyone looks up in the sky. And for the first time, a man-made thing is floating in space and that's Sputnik. The fact that the Russians beat us to the skies with Sputnik kicks off, I think, a level of panic from a geopolitical standpoint that's hard to connect to today. And so after the panic of Sputnik, Eisenhower creates the advanced research projects agency or ARPA. And in those years, ARPA was pouring money into space, into missiles, into computing. And we wanted to ensure in this nuclear-armed world that weapons could not destroy a centralized command system. And so the network itself needed to be decentralized. This need to ensure reliability. Combined with a slightly more mundane problem that a one-bobbed hailer faced when he was at his office in the Pentagon, which is that he was working at three terminals. He had one machine connected to MIT, another connected to UC Berkeley, and a third for a different research system altogether. And he could talk to any of them, but never at the same time. So he's rolling his chair back and forth. And he quoted saying, "We have to find a way to connect all of these different machines," he told his boss. And after a 20-minute meeting, Taylor walked out with a million-dollar budget and a simple mandate to make it happen. And we won't go into every moment and step because the ARPA net itself is quite a story. But I think a couple of innovations set the stage for the internet in a really important way. One is the concept of packet switching. If you think about a phone line, AT&T built its whole business on this idea that you're going to connect a phone call between two people and there's going to be a circuit that connects those two phones together. This is called circuit switching. But there's this crazy idea, which is if you want the resilient system, so that if one node goes out, the next thing doesn't go out, you need to route things around in different ways and not be dependent on the one route you have. So you need to have more of a spider web node to say, "Well, from here to here, there's actually five or ideally 50 or 5,000 ways to get between those two points and to do that you need a flexible communication system. And so the idea is to break up information into packets, little pieces of mail that would get sent from one node to another, different packets could even take different routes. But on the other side, someone would reassemble them into the message. It's the same singular message broken up into many, many different pieces and routed through many, many different pathways. It's fun. I hadn't thought about this until we connect it to the time sharing story, but it's actually about better utilization of the network. In a circuit switch mode, you have this direct line, you and I have a direct line together, and you have a direct line to someone else. That line is usually empty, right? It's pretty hard to fully utilize and build out the circuit switch thing because the phone call ends and you're like, okay, now it's just sitting there and used. In packet switching, packets are finding their way through this wild world. So a lot of skepticism obviously from AT&T, which said if you want to have two computers talk to each other, we'll build the phone lines and you can run them. But a crack team that had spun out of MIT, this company, Bolt, Baroneck, and Newman, latched onto this packet switching model and ran forward. And the core thing that they needed to build was what was called the interface message processor or the IMP. This was ultimately the router that would sit in front of the host machine, which was the university MIT Berkeley would have these main frames, but to connect those machines together, there would need to be these nodes that would sit in front of them and be able to build this network. And so BBN used a Honeywell mini computer that was a fridge size cost about $80,000 and built the logic around packet switching and rolled this out. There are a lot of engineering heroics that went into getting the first machine ready. But then it was time. Almost 55 years ago, a small group of grad students gather at UCLA to wait for the machine to be rolled off the truck, champagne in hand, and are able to celebrate the arrival of the first IMP, the second one delivered to Stanford weeks later. And the prophetic first message as Laura has it was meant to be login, right? You now have users that can log in. But after putting the first two letters, it crashed. And so the first message was simply low. Low. So close to low. And of course, you know, what is the network with two connections pretty limited, but then they rolled out UCC and a Barbara University of Utah. And then they were just rolling out new IMPs to the large universities month over month. And these became nodes, almost a new node every month. And then every new node that comes on the network amplifies the value, right? Because now it's not just by directional communication. It's multilateral communication. So they're building all this logic to do the resilient routing, to do the discovery of devices and everything that you need to do that. And of course, the original pitch was, hey, we've funded as a government all these expensive computers at all these research universities. Let's drive utilization. Maybe one team has a special program for doing something, another team has a special program for doing something. Let's let the departments connect one unexpected application really bubbled to the top that the researchers discovered quick asynchronous messages were often more valuable than logging into someone else's system to run the code. And in 1972, Arpa's Bob Khan said, you know, everyone really uses this thing for electronic mail. And the network's purpose had quietly shifted from sharing machines to connecting people in the form of email. It's wild how as soon as you have enough connectivity, it's always the killer app. Right. Because it was always people communicating at a distance. That's right. People wanting to do what we are biologically programmed to do, which is connect with other humans. And as sophisticated as these machines were, it was the simple act of asynchronous communication that became the killer app. And more people wanted it. There became new ways to connect within your kind of local network over these terminals. So then you could connect into the broader network. It was no longer that, hey, you want to access to a terminal to access the machine on your campus. It was you wanted to terminal. So you could email your friends at the other campus. Sounds like an early social network that we'll get to later. And so the expansion continued. By 1972, there were 29 nodes by 1975 over 50. And some of these connections weren't just from university, university. We were now leaping oceans. It was clumsy, but we were able to go over the Atlantic Ocean from a node in Virginia to the one in Cornwall, England. There was some story where they came back from some conference in England and someone had left their electric shaver behind. It was middle of the night. So in England, it was three o'clock in the morning. But he knew that his colleague was a workaholic. So he sent off a message at three o'clock in the morning, England time seeing if this person was awake. And he was. And he saw they used log on. He's like, hey, did I leave my electric razor there? Can you get my razor, please? And it all worked. It was phenomenal. So by the early 1980s, other research networks started to pop up in the US and otherwise. People like, oh, this arpinette thing is pretty cool. We're going to build our own. There was decknet. There was an IBM net. There was various nets that companies started to stand up. Other research groups had a new challenge of we want to talk to everyone on any of these networks. And how are we going to do that? And these IMPs were not designed for that. They had presumed that there was only one network that they were trying to build out. And so Vince Surf and Bob Khan, who were involved in arpinette figured out what they needed to build was this common language to figure out addressing, where are we going to send to this packet of mail? And then also the actual control protocol for the actual sending of information to make sure you don't send duplicate packets. And so this became TCP/IP, TCP for the transmission control protocol, and IP for the internet protocol. You can think of IP as solving the addressing problem, right? You might be familiar with your IP address that then communicates through routers, what you're looking for, who you're trying to get the packet to. And then TCP is the kind of reliable way that a packet gets acknowledged. Hey, this little chunk of information has arrived. It's like the certification of the mail signing for my package. I have an address so the package knows where to come. And then I sign for it. And I don't need you to send me the package again, because I got it. And so this combination became the backbone for connecting all these networks together. And ultimately the backbone for the next stage of not the arpinette, but the inter net between networks, inter net. And so in 1983, they took every arpinette host and said they need to move to TCP/IP. And that switch, January 1, 1983, became the framework for the modern day internet. Many networks all speaking the same language able to connect. And that's where we have the advent of the dot com, the dot edu, the dot gov, because we had now standard protocols for multiple nets to be communicating together. 20 dot 3 dot 1 dot 72 doesn't have the same ring as pets dot com. And important one. But we're not quite there yet. No. But by the mid 1980s, as Ben mentioned, there's multiple nets now connected. And arpinette itself was showing his age by 1990. It became apparent that the arpinette needed to be decommissioned. But the work it had done, the ideas of decentralized networking, packet switching and open protocols had firmly taken root and research institutions and these large organizations had firmly established how they were exchanging messages. So they were able to send files back and forth, maybe send each other's research papers and data sets. But this was not a thing with the web browser yet. We needed a structure around documents. And so Tim Berners-Lee, a researcher at Surn in Switzerland, proposed the document structure that would become the application layers infrastructure of the worldwide web. And so the network that started off as a Cold War research project in response to looking up at the sky and seeing the Russians on Sputnik had now become the public and commercial internet and open highway for data and the foundation for which we're going to build the modern data center economy. So we now have this expanding network. But the computing on this network was still this terminal-based interaction between a terminal on a mainframe that then connected to the network or terminal in a mini computer to the network. And I think it's worth saying a bit more about mini computers, which were both what became that IMP device and then was also a primary source of connection. These were not something that you'd buy for your house. They still were the cost of maybe a car when they first came out. But they were much more accessible than a big IBM mainframe room-sized computer. And so the company Digital Equipment Corporation or DEC launched the first mini computers in the mid-60s. A lot of what innovated on the mini computer was software like UNIX and C programming and sockets and all these foundational innovations that would later echo through to today. But there was room for a computer that a normal person could buy. On one fateful day, January 1975, the cover of Popular Electronics Magazine showcased the Altair 8800. A machine you could buy at home for $439 or $1,500 souped up and run your own code. And there are a few hobbyists that went to the newsstand and picked up this magazine. Two in particular saw the cover and decided to do something about it. And so Bill Gates and Paul Allen, famously when they were at Lakeside High School in Seattle, they had access to a computer and terminal and they would spend hours and hours programming. But that's different than having one in your own house that you can play with. And so when they saw this come out, it was immediate that this was a moment. that has gotten cheap enough that a normal person could afford to have their own personal computer or PC. And you could say that if the first one was released at $450, it was only a matter of time before that would become accessible to more and more people. Microsoft formed immediately to sell a basic compiler for this device. The Apple II launches in 1977, which kicks off the PC way for many folks. And it pairs with VisitCalc in 1979. So all of a sudden, you had a killer app of a spreadsheet that wasn't just for home use, but now had like a corporate function to go and buy these PCs. And so someone can model a budget without needing to connect to the mainframe down the hall. I can stay at my desk and run my finance operations. That's right. And IBM, while they weren't the first here, they're actually pretty quick to realize that there was a problem for them. If everyone is doing all the computation on their desk, they're not going to need the mainframe down the hall anymore. The glass house has been shattered. [LAUGHS] And so they kicked off a skunkwork project. It was actually pretty amazing. They isolated it outside of the New York region down in Boko Riton. And they said this team is going to use off-the-shelf parts to design an IBM PC. And so in 1981, they built and launched the IBM PC. And they actually licensed MS-DOS from Microsoft, founded six years earlier, as the core operating system. They thought that hardware was the business. Time would show them to be incorrect. And a bunch of IBM compatible devices, including Compact and others, flood the market through the '80s. And so prices fall. Hardware becomes commodity. And Microsoft, with DOS and then Windows, becomes the windtel duopoly that ultimately takes the mind share and market for my IBM. It's amazing. You've got Lotus 1, 2, 3 dominating spreadsheets. You've got WordPerfect. I remember using that for word processing. And then Microsoft comes along and says, I'm going to bundle all this into office and really takes the cake. And so productivity is through the roof. And we're able to do more personally than we could ever do before. And the way we collaborate, it's that magic floppy disk. We're able to hand that back and forth and keep on rolling. And that's great, right? Until it's not. Until it's not. You want to work with someone across a big building? You're going to run them a floppy disk. And what if they have a different version? It's a mess. I think the personal and personal computing started to become a hindrance here. And so we needed to connect these devices as well. Enter the invention of Ethernet and IBM's token ring that wires the floors and connects these PCs into a corporate network. And even new set of companies, you have Novel-Billy Netware that actually turns a server into a hub for shared disks and printers. You have a PC on your desk, and there's maybe a printer room. And you want to go print to that. You'd have a different computer that is a specialized PC that we're going to call a server that's connected to that printer or connected to the shared storage. And now we're able to access files over a network. We're able to print over a network. And a whole company, Novel, that at some point in time was actually the second largest software manufacturer after Microsoft was booming in this era. Now Microsoft does what they do very well and responds and builds the network-connected operating system, eventually Windows NT and other services so that you can do it all within Microsoft's ecosystem. So throughout our conversations, a lot of the OGs in this industry pointed to this moment as the introduction of the client server era. What does that mean? It means an application is always split into you had your computer sitting there on your desk running as the client accessing an essential database or a server in the back room. Now, a bunch of technologies came up to support this from Unix servers to ERP systems, to Windows NT. But now all of a sudden, you would think about an application as networked from design, where you'd have this local client software that could be using the best capabilities of the local PC connected to the server, not across the world, but the server in the server room on site. What is that server down the hall or in the other room or across the world? I think most notably is that in this era was the era that you take the architecture of the personal computer. So this is the X86 kind of Intel-based architecture or the Sun and Workstation architecture. And increasingly, those would become the servers that could run these applications. This was a notable shift again from the IBM Meanframe era. And it got to the point where someone selling your company an application would sell you an appliance, which was really like a package of the software and the hardware together. And so you ended up with the sprawl of different appliances that IT is managing, all serving different functions, all being written with their own bespoke software with their own operating systems. And so while the functionality was amazing, people got very inefficient. Up until this point, there was a culture and an ethos amongst data processing professionals that put conservation as the highest level of ethics. To waste a CPU cycle or a byte of memory was embarrassing. I think part of it was it got cheap enough. These servers got cheap enough. These same servers would have been $10,000 a month to rent, just 10, 15 years prior. And now it wasn't unreasonable to have them sitting there idle. At the same time in a macro sense, this inefficiency later creates the space and value of the evolution of time sharing and virtualization. Where we say, why can't we run these applications on one box? Why do they all need their own box sitting there waiting for a command? That's right. If the box is not being maximally utilized by a single client, share the box. We will do a whole focus on how that comes to be. But this is still an era of deep innovation, particularly on redundancy. You have rate arrays, which is redundant storage for backup, tape libraries that are off site. Because you still have these massive points of failure. Your company's server, which might be your system of record for your customers, is sitting there in the closet. I heard from someone who ran some of IT at REI at a point that like they had a flood near their headquarters and like, well, the website's going to go down. We're going to lose all this business. And so this model was not designed for scale and redundancy in a major way. So to make this moment concrete of just how computing power and connectivity was starting to shift, let's talk about a deeply innovative retail company that no is not Amazon. We're talking about the OGs of major retail and scale. It's Walmart. That's right. And so you're at the register at your neighborhood Walmart location. You scan the toothpaste, the barcode beeps. Within seconds, a satellite dish behind the store sends that transaction to the sky and over to Benton, Villar can saw where they're hosting their mainframe computer, record the sale. A few minutes later, a massive warehouse with data would update how many tubes of toothpaste you would just bought. And then perhaps before the end of the day, Procter and Gamble's factory would receive an update that they needed to make more toothpaste. This is the cutting edge of retail in the late '80s. And Walmart makes this massive decision to take this a step further and really drives innovation across the retail industry. They invested $24 million to build their own private satellite network, linking all Walmart stores to headquarters. This was really fairly unprecedented at the time, right? It is the largest private satellite network that had been built. And it created a unified real-time machine. And so any single event that happened within the Walmart ecosystem rides on this private network, and it enables them to mine their data in a way that was unheard of before. By mining their sales data, which now could be collected in real time across all Walmart stores over their private satellite network, they discovered that when hurricanes approach, the sale of pop-tarts increased 7x over their normal rate. And having spent a summer working in Bentonville, Arkansas, I can tell you this is deep in their DNA. They are constantly looking at signals like this. Specifically, they discovered that it was the strawberry pop-tart that was most in demand before a storm, which led to the legendary insight of meteorologists predicting a severe weather event, and Walmart stalking their effective stores with pallets of strawberry pop-tarts. - It makes you wonder if the reverse beam pulse will, it's like instead of checking the weather, you go to Walmart to see if the strawberry pop-tart is there because it's such a reliable system. They're like, "I know that they're watching the weather." But I mean, there's some crazy innovation here, both the satellite link and then Walmart supposedly was the owner of the first commercial one terabyte enterprise data warehouse. They built it, they call it teradata. And so they were just maniacal about making sure all the sales and customer data flowed into one place. By 2001, just nine years later, that one terabyte warehouse had grown to 70 terabytes. - Wow. - And so this is just an explosion of connectivity inside the enterprise. While there was the personal computer, and some people had fun with their computers at home, I remember playing on IBM XT and as a kid, and some early programming, early games. It was not like most people had the real personal use case. We're still talking about a corporate centered world. So we talked about the rise of PCs and the growing client server land inside of a company. We talked about these data centers, we talked about Walmart satellite network. But what's happening now going into the early 90s in this whole ARPANET and SF Net Internet thing? What's having outside the office? - Yeah, and perhaps what's happening under the ground? to the first network. that was available to all these researchers was the evolution of this into the NSF net. And it became the de facto US internet backbone. This net was connecting 2000 computers in 1986 and expanded to over 2 million by 1993. And it was no longer just researchers who wanted to do stuff with it. And so the design and topology was this high speed national backbone that had at this point gone to a T3 line, which is at 45 megabits per second, that connected a small number of regional networks. So kind of like a central hub and spoke model. And then these regional networks would connect to universities, labs, nonprofits. But this was not designed for commercial scale. In fact, it was prohibited, according to their terms of use, to have commercial traffic running on that backbone. It was specifically designed for research, education, and government data. So I think some private commercial network could happen inside of just like a regional hub, but not across that whole big backbone. So you did not have the beginnings of what could be a commercial internet. If two regionals, you know, New York and Philadelphia wanted to connect, they had to flow back up to that NSF net backbone. There was no neutral point where they could connect in exchange. And so this is the moment when commercial ISPs and telcos started to see, okay, like this internet thing is interesting. This packet switched model. People want to do new things with this. Maybe it's trying to get out of the lab into commercial use. How are we going to connect? What are we going to do? And in 1992, a group of network providers were sitting in Virginia drinking a beer and they decided to connect their networks outside of Tyson's corner. Now this specific group of engineers were from Metropolitan Fiber Systems, the local telco. They chose Tyson's corner outside of Washington, DC because you had a dense network of defense contractors and early providers, which were heavy users of the current internet. And they famously set up in a repurposed parking garage to become the de facto on ramp for new ISPs. Now when you think about an important hub, you wouldn't typically think of a parking garage, but it was the right place at the right time and it turned into what's called Metropolitan area exchange east east coast. The May East is what formed. And if you connected into May East, that meant that you had the internet at your doorstep. This became the hub for the internet. If someone sent an email from London to Paris, it most likely went through May East. Within a couple of years, you had roughly half the world's internet packets flowing across the May East parking garage. And I think this is something we'll see again and again, which is that the internet forms just around hubs. And it's not always obvious why that became the hub specifically, other than like it did first. And there's just this gravitational pull of connectivity. And this wasn't new to the internet. This was something that we saw with Tocos. There was this concept called carrier hotels. We're in a city like New York and you had two different carriers that are trying to connect with each other, right? Think sprint and AT&T. Instead of having to connect all throughout the city in multiple spots, they would all show up in a neutral zone called a carrier hotel and build their connectivity infrastructure there. You'd be able to tap into each other's long haul routes, their local fiber routes without having to build their own intercity footprints, each of them individually. And so May East was the first sort of flavor of this where they would all come in and connect to ultimately the vice think a switch that is connecting. Okay, you know, you're coming in, plugging in your ISP traffic here. I'm putting in mine. Now users across our ISPs can connect and it's just one big switch room. The problem is the internet is scaling and you don't necessarily want to all be bottlenecked on one switch. It's not the most secure thing. And you know what? If you're building a video streaming service or you make a deal between two ISPs, you don't necessarily want everyone else to be in on that deal. And so you saw the evolution of this model, two different model that became known as meet me rooms. And these are neutral physical rooms where an ISP or someone trying to hook into an ISP can provide their boxes and their connectivity. And then those two can connect together directly. So I'll bring your own box method. So instead of everyone connecting through the existing box, you BYOB your box for the deal that you want to do in the private meet me room. That's right. Something sounds funny about BYOB to the private room. But we're talking about ISPs connecting for data. The farthest thing from a non-plotonic conversation as you could be. And so sure enough, this all worked. And the NSF kind of officially sanctioned this method and designated these NAP points, network access points. The first one, Bain May East, the designated sprint to run at NAP in New Jersey near the transatlantic Kibble landing points, one in Chicago and one in San Francisco. Then they eventually added May West in San Jose. And these meet me room models started to take off the carrier hotels. You had one Vilt Wilshire, a large building on the west side of Los Angeles, which were law offices and gave way to a single floor that could host hundreds of carrier routers and thousands of cross connects, eventually making it one of the most valuable space per square foot on the entire West coast. It's so funny. It's like this ugly building or just relatively nondescript architecture. You're like, what's going on in there? This is where the West Coast internet is coming through. That's right. And we'll talk a little bit later about undersea cables, but they come in and want to find their shortest path to one Wilshire. And the business model was pretty genius for this. These telco hotels, these meet me rooms, ultimately provided power, cooling and cross connect and they would charge rent and the dot com boom to come would boost this model to new heights. And it provided an elastic infrastructure that internet companies in the dot com boom and after could leverage, including as we'll see soon, hyperscalers where you could flexibly increase and decrease your capacity because they were specialized in providing all the necessary infrastructure to be able to host the connectivity. So all of this infrastructure being set up to commercialize the internet to provide a scalable backbone to enable the private market of ISPs and telcos and others to invest in making the internet faster. And we now have the worldwide web. The NSF has actually funded a little project called mosaic, which is the first kind of user-friendly web browser where personal computing enter the mid 90s and the dot com boom. The ISPs are ready to build the network. All we need is the users. And boy are they coming. May of 1995 Bill Gates writes the famous internet title wave memo. The internet is the single most important development to come along since the IBM PC. And it's that same year that net scape when public. Why was net scape so significant then? I think significant for two reasons. One is a kick off the accessibility of the internet. It turned to the internet from this network for researchers to share files to this browser that you could download and install and access the web and open it all of that up. And then it was that from a product perspective, but that also captured the economic perspective and interest of is there a thing here? Is there a new industry at boom that you can make your millions or billions off of from founding to IPO in such a short cycle kicked off a mania? It kicked off another mania in the business world, which was having barely any revenue or barely any profits. You can hit a multi billion dollar valuation, which is what they did. And the web exploded. You had 23,000 websites in 1995 to over 10 million by the year 2000. And global users climbed to over 350 million. Nasdaq tripled in two years. If you put a dot com at the end of your name, just kind of like you put an AI at the end of your name today, you could raise millions on an idea and a side deck. And in 1999 alone, you had more than 400 internet companies going public, pulling in 40 billion dollars. What a liquid IPO market that we could only dream of today. Microsoft hits all time highs in the stock market. The energy was manic, right? Founders in their 20s would become people millionaires overnight. Engineers were hopping jobs for stock options. This is the rise of the air on chair and the foosball tables and the new economy were the rules of things like revenue no longer applied to business. But it wasn't just applications. It was infrastructure as well. Carriers spent half a trillion dollars on fiber and wireless. You had these co-location companies expanding at breakneck speed. This company called Exodus Communications was the world's largest web hosting provider at the time, providing server co-location. And as revenue went from 12 million in 1997 to 250 million two years later. And it peaked at a $32 billion market cap three years after that. It was laying the infrastructure in the ground and building the applications above it in a period of unprecedented growth. And so the launch of startup, you had to build a site, you had to build a service, you had to build a database, you needed money way ahead of time to buy the servers to stick in the co-location box. Wait, it's your saying in order for me to launch a web business. I had to buy hardware. Exactly. You had to take most of your venture capital dollars and spend it on servers. Even before you knew it. if anyone wanted to go to your website to begin with. So you couldn't test your idea out. You couldn't AB test. You couldn't do a landing page that drew in a wait list. Couldn't do any of that. What a different world. So all these new servers are trying to run on the new backbones that are being laid to power this boom. And the traffic still at this point meant at only a handful of public exchange points. Networks were plugged into these shared boxes at places like Mayease that we just talked about. And that can only scale for so long. You'd start to hit choke points. And Mayease, one of the earliest network access points, became one of those major choke points. We needed different models for companies, ISPs, non-ISPs to be able to connect. You had deck kicking off Palo Alto with the Palo Alto Internet Exchange, a non-Telco neutral spot. Then you had the founding of Equinix where they took that model and scaled it. And with their first site in Ashburn, Virginia. Ashburn is the Wall Street for data centers. In 1999, Equinix launches their first data center under this new model in Ashburn, Virginia, right next to Mayease, the choke point but the original network access point. Isn't this near like DC? I've never been there. Have you been to Loudon? I grew up right outside of DC on the other side of the river. Loudon County is on the Virginia side. It's rural farmland, it's past Delus Airport. There's really nothing there, but it's proximal to a large East Coast population, the intersea cables, and Mayease. And you actually had AOL choose Loudon County in the mid 90s to set up a huge dial-up campus. And so they laid fresh fiber and drew even more carriers into the region. And Mayease had grown so large, it outgrew the parking garage that was in. And so the exchange relocated to Ashburn and so you've got this unassuming farmland outside of Delus Airport. And it was really catalyzed by a couple things outside of AOL pioneering the new site. It was policy led. So Loudon County ruled that data centers could be treated like ordinary office parks. They just eliminate all these special use hearings and provide incredible tax breaks over time to attract data centers into this network. Coupled with that, you have Dominion Energy to kind of see what's coming ahead. And offered some of the lowest industrial rates to string high voltage lines to this empty land, to bring power and fiber together to create the new data center model, which Equinex pioneered in Ashburn and became the largest internet hub on planet Earth. - So it was the combination of perhaps for a moment, cheap land, not so much anymore, fiber and connectivity, cheap and accessible power and favorable policy. One interesting policy story that I heard on this, Apple was looking for where to put a new site in the late 2000s, 2009. Apple's obviously building more services, needs more storage, building more data centers. So they run a process and it turns out that North Carolina gives them a better deal. So Virginia fights back. And in reaction passes major tax breaks to say that if you're building a data center, basically if you're building anything more than $150 million of investment, you're gonna employ more than 50 people, no tax. - Which you, of course, are, if you're building a data center, of course. Yeah, you can build a data center for $150 million. - Local policy, as you mentioned, is so important. Federal policy actually played a big role here. In 1996, we passed the Telecommunications Act. And one of the main things it did was force the incumbent telcos, which were regional monopolies from the AT&T breakup to lease their physical network, their cop repairs, their fiber to their competitors. And so prior to this, a data center was a private enterprise tied to a single carrier, because the carriers only use their own fiber. Now it enabled carrier neutral sites. And so you could become a tenant of a data center and choose from multiple fibers provided into that building. And it opened up an explosion of choice for tenants and for this care and neutral model. - And so this is a flywheel of a deregulation environment to build us, 'cause once they have the tax incentives in, it becomes a major source of economic prosperity for the region. Now, we'll fast forward later on in the story to today, and maybe hitting some of the first real pushback. - And similar to Frankfurt, Amsterdam, London, Tokyo, the biggest hubs are where you can find cables and carriers and connects the most networks with the least amount of friction possible. And it's this flywheel that continues to make Ashburn the largest home of data centers in the US. - And what a time. In the first six months of 1995, internet traffic was doubling every 100 days. The telcos were convinced that you couldn't overbuild. You just needed all the fiber you could put down. - World common and these other telcos, they just, they poured billions into this. And the idea of overbuilding, not possible, particularly in this internet boom, right? But by 2000, how much of that installed fiber was actually being used? It was only like 3%, it was all it out, 3%. So they just laid down fiber, the shards of glass that are just sitting there, empty. - So to give you a sense of the amount of fiber miles laid during this period, you could go around the circumference of the earth 5,000 times. - Wow. - And so this was all over land, right? But how would you connect to Europe? - Ah. And that overbuild, your right, was not just limited to land. Nearly all intercontinental internet traffic rides on undersea cables, which I'd kind of heard of, but I didn't really have a full appreciation for the fact that bundles of glass threads are wound together into a garden hose structure and laid down on the ocean floor all over the world. And so imagine a planet stitched together with hair-thin strands of glass tucked into an armored hose and laid across the darkest parts of the ocean. That's the undersea cable system. And it's the real physical internet that connects the continents. So 99% of international data still rides on these cables, not satellites, that's racing pulses of light through fibers thinner than the human hair. - The story arguably starts way back in 1850s when we had the first telegraph cables that were crossing the Atlantic. They brought those across steamships and landed the first link in 1866, able to send news for the first time across the ocean. So instead of weeks, you could get the news in minutes. - Yeah, you're not sending the news via ship. You're sending that via electrons. - Amazing. - And fast forward to 1988 and we land the first trans-Atlantic fiber optic cable. It runs between the US, the UK and France. And it kicks off this new era of cross continental capacity. By the late 90s, as this boom was happening, you can imagine the funding routes going into wiring all of this up. - You lay all of this cable down with specialized cable ships that surveyed the seabed, unspooled the cable, and then near shore, they bury it underneath the protective from anchors and storms, and then rise it up through some non-descript concrete box. And this is, this garden hose has optical repeaters that boost the light every 50 to 100 kilometers. So you can sprint thousands of miles without fading. And then you get to land and these concrete bunkers have the cable hop up into terrestrial fiber and then run straight to your one-wilshur or any of your nearby hubs that then connect you onward. - What a wild thing. You have a garden hose-shaped thing like moving and all this dead light. What do we actually mean? Well, modern cables, an example, one laid in 2018 can move 250 terabits per second. To conceptualize that, you can send 6,000 HD movies in one second or about 20% of global internet traffic can go in one garden hose. How is this working? Well, at the beginning of using fiber optics, you would shine a laser down and blink your zeros and ones. But we've moved from doing that with one wavelength to doing what's called wavelength division multiplexing. That's a fancy way of saying we use the rainbow. We're using multiple, usually it's around 80 to 120 different colors that can go down the same cable at the same time. The other thing is we've added more fibers. So instead of there being a pair of fibers, we now have up to 12 to 16 pairs of fibers. And then the last thing is called coherent optics. And this is a way of modulating the amplitude of the light. So instead of it just being on and off, you actually can have multiple steps. All of this adds up to today, probably 250 to 300 terabits per second maximum capacity in one hose. There's hundreds of hoses around the world. And I think the other thing that's really cool about this is the thing that was laid was the glass. We keep increasing the capacity of the glass because the glass is the glass. Resilient infrastructure is kind of like railroads. It's like we're still using the tracks from long ago. Even if the engine gets upgraded. And just like we've been talking about with this network effect, when you bring more networks in, it increases the value and the speed. And so that's a big reason why Ashburn, as we talked about, hardened. Once hundreds of carriers and thousands of cross connects land into a single place, moving it is impossible. And so if you need to reach Europe fast from the East Coast, you're going to co-locate where the undersea cable is already coming. You've got New York, New Jersey, Virginia, tying to Cornwall, England, and to Merci. You've got the Red Sea and the Mediterranean cord are connecting into Djibouti as a critical touchpoint. You've got Miami as a key touchpoint into Latin America, Japan, Singapore, Hong Kong, Taiwan, or key corridors in Asia Pacific, Mombasa, and. and Legos, now light up Africa's East Coast. So these are just hundreds of cables have converged that have connected the entire world and this then forms the network of data centers that we have built and will continue to build through this story. And so the boom continues, right? No. There was something like the bust to that boom. That's right. And by 2001, everything collapsed. Advertising folded, startups folded, Exodus, we were talking about earlier, filed for bankruptcy, had nearly six billion in debt. PSI net, one of the largest PS had already collapsed. And so was this the death of the internet and the death of the value of all of this fiber that was laid? Clearly not. It was the death of a particular moment and an overbilled in a bubble. But in fact, the actual infrastructure that was built out would prove immensely valuable as services that mattered matured and business models matured. In other words, I'd say the application layer of this era died, but the infrastructure lived on and would eventually thrive. So the dot com tide went out, but the overbilt of assets were exactly what we needed for the next chapter. And it included infrastructure that we can't live without. The carrier hotels, the fiber, the data centers, the glass in the ocean didn't disappear. It just changed owners. And there was a fire sale, right? Assets were being sold at a fraction of the cost. A few key actors survived and a few new ones stepped in. Equinex survived the crash. They doubled down on the real asset that they had, which was interconnection. That neutral meet me room turned into a marketplace where competitors paid you to be neighbors because the value was in the speed and the reliability and the flexibility that these carrier hotels provided. Meanwhile, private equity swoops in as they will in every bust cycle and reframe the category. So one in particular buys a couple dozen distressed facilities around the world turns it into a vehicle called digital reality trust. Takes a public as the first pure play data center reate and it treats compute space like real estate. So it wasn't high tech, moon shot assets they were buying. They're bringing patient capital in to standardize the shell, finance it cheaply and get long leases, which becomes a blueprint for the next two decades of data center buildouts. In addition to the sobering environment and economic environment of 2001 from the bust, there was also September 11th. These financial institutions were still operating in this moment of having their key servers, trading information, connectivity with banks in their offices or right near their offices. So when 9/11 happened, the Verizon 140 West Street Central office, one of the largest telecom hubs in the city was blasted with debris and dust, flooded it in the equipment rooms, tens of thousands of voice and data circuits were knocked off line immediately. Most of the circuits were powering exactly that brokerages, market data providers, the trading floors, low latency connectivity to the stock exchange and clearing houses. So you imagine the market is just disconnected now in a flash. And this cascaded engineers worked night and day to bring things back online. Was looking into Morgan Stanley's experience. They saw their whole trading system go down. They had a disaster recovery site in New Jersey, but they didn't have the same level of connectivity and data feeds. So took them a few days. They ran new fiber through building basements, patched hubs into another telco hub that was still operating to restore capacity so they could trade when the stock exchange opened September 17th. And I think coming out of this, there is a whole reshaping of the data center world to think about resiliency in a new way. I think it showed the physicality in the city of this connectivity, right? In this moment where you could take for granted, that you could take action on data over there, whether a trade or a market data. It wasn't enough to have redundancy on another floor in the same neighborhood, right? We had to start thinking about an infrastructure build out in different locations with different networks, facilities and routes to really build true resilience and switch over to the point where now for data center goes down to this automatic rerouting. And we don't see those same blips, although it happens from time to time. Just early 2000 phase is a real maturing and growing up of the entire industry to realize that these servers and data centers are holding important financial data and need to be treated as such. But meanwhile in consumer land, there's a glimmer of light and it's a big one. This is the era that we move from that squeaky, squealy phone modem to broadband. I remember touring for colleges and some had Ethernet across the campus and others didn't. I feel like getting broadband to our house was a radically different experience. It was like a different internet. The image didn't load from top to bottom. You're just fly through. And so this is when BitTorrent starts soaring. This is when Skype launches 2003 and you can actually make VoIP calls. Napster was possible. World of Warcraft launches 2004. I remember that taking the college campus by storm or early web video products started to come out. By 2005, you had a billion people online, about 16% of the planet. So all those folks prognosticating with excitement in the late '90s were not wrong. They were just off by five or six years. And so this was also the advent of CDNs. So, Akamai's footprint exploded to provide more and more storage and replication and caching at the edge. And so what exactly does this mean? It means at the places where your ISP is connecting of someone's downloading an image. Let's say you load the New York Times and your computer and someone else does on their computer down the street. You don't both need to go all the way back to the New York Times home server for access to that photo. It's now been cached on a nearby CDN that's directly hooked up to your ISP. - So then the early to mid-2000s, you have the consumer coming back. Applications are flourishing. The internet is becoming a part of the fabric of society. And they're a handful of companies that survived the bus and captured this moment unlike any other. And they not only built incredibly large consumer and enterprise businesses, but they actually became critical infrastructure companies that helped build the modern data center world. That is what we're seeing booming today. And so the place we're gonna start with is the best place to buy books, Amazon.com. - So whatever book you wanted to find from A to Z to line 1995, Amazon launches by 97 A IPO and by 98, they are no longer just a bookstore. They're on their path to becoming the everything store. Now, it was not initially, basis and Amazon's intention to become the infrastructure provider of the world. But this high growth moment of the late 90s set the stage for what they would need to build not just for themselves, but for everybody. In those early days, they were running expensive quote unquote reliable servers from the likes of deck, extremely expensive products, high margin servers. Now the problem is, Amazon was not a high margin business. They're trying to go for scale. They're selling things at whatever the cost was to pass through. They are a retailer trying to be the lowest margin retailer out there the cheapest way to get your book delivered to your doorstep. - So running a retail business, they're always tight on cash. - So to spend it on servers, stopped making sense. So by 2000, they're spending so much an infrastructure, they're worried this was gonna bankrupt them. And so they kicked off a big project to rewrite all of Amazon.com on Delinux and to run it on much cheaper HP servers. And this is when Amazon was famously a huge monolithic code base. Every new category they launched, they had to work across their entire code base and it became this terrible. - By I think around 2002, Bezos had had enough of that and he issued the famous API mandate that internally every team had to expose functionality through hardened, documented service interfaces, designed not just to be used by internal teams, but eventually potentially externalizable. Classic phases, there were no exceptions. Every team must communicate through these interfaces. There was no back doors, no direct threads, no direct linking. Doesn't matter what that technology did, it would without exception be designed from the ground up to communicate externally to other teams. And if you didn't do this, what would happen? If you didn't do this, you were canned. So in this moment of reboot, it seems like the question that the company's leaders are asking the sales is, how are we able to scale our business like a software business and not like a furniture business? What if compute could scale with demand? And if we can do this for ourselves, why not rent it to the rest of the world? They were on the precipice of not just a technical breakthrough, but a business model breakthrough. Because for decades, running an online business meant these multi-year leases, these expensive servers, over-provisioning to handle peak demand. And Amazon would go on to flip this on its head and fundamentally change internet businesses by saying you can rent a server by the hour and pay for only what you use. It is both deeply innovative at the time and also finding you because we've had decades, century of doing this with our electricity bills and our own houses or water bills, right? This is utilities. utilities you've always just paid for what you use. But what it continually enables is the driving down of cost, better utilization of centralized infrastructure. And Amazon had just lived through this painful period of having to rewrite their software and change their server architecture. And I think it was two things. One is we never want to go through this again. A, B, no one should have to go through this again. And C, if we start to build the infrastructure for the world, that's going to accrue to our costs and our benefit. And there's going to be a flywheel here, just like any scale economies provider ever experiences, the bigger we get the better. That's right. And that utilization is such a key point. So, so what happens in March of 2006? Amazon launches the first real AWS service, S3 or simple storage service. What does this simple storage service do for me as a small internet business? It lets anyone put a blob of data on this non physical disc and access it anywhere in the world. And that sounds simple as it is in the name. But it was shockingly hard to put a blob, whether that was a megabyte or gigabyte or a terabyte of data out there and have everyone around the world be able to access it quickly. Amazon abstracted everything away so that you could do that and just pay a monthly fee. You didn't have to build a server and plug in a hard desk and build another one that copied the data over and all these other things. It just gave you what you need as a developer. So this enabled me to store information. Correct. A few months later, what did they do? They launched the elastic compute cloud, also known as EC2. So I can store and now you're telling me I can compute. How does this work? You can compute. And so what EC2 essentially was was the ability to spin up computers, servers, as you saw fit. Now what they were actually laying you spin up is what's called a virtual machine, where you can say, I want to run Linux or I want to run a Windows. I want to run some SQL server OS. And I would have this virtual machine where I could deploy that run it and run whatever code I need to and if I need a second machine, I push a button and get a second machine, me a third machine, a third machine. And I pay by the hour by the machine only for what I'm using. This is outstanding. So I'm building a business and I think I'm going to grow fast. But I don't know what traffic I'm going to get next month or in six months. And so I just raised a bunch of VC money. Now you're telling me I don't have to buy these expensive HP and sun servers. And as I grow, I can just rent more compute and rent more space. Yeah, not just that, but I think the activation energy here was brought way down before there's the challenge of getting a server into space and all that. But like in this model now, you just put down your corporate card and you're often running with the foundational building blocks that you need. This is the Cambrian explosion for startups. Key to this is the utilization point. So many startups have bought servers that then never hit full utilization or we talked about all these appliances sitting in the backroom closet, not in full utilization. What enabled Amazon to drive utilization? It was the fact that yes, they gave you a quote and quote server to run your operating system on. They did not give you a server in reality. They gave you a virtual machine, not a machine, a virtual machine. But what is a virtual machine? A virtual machine is the flavor of the concept of a plan, the concept of machine. And this goes back to a company VMware that was founded in 1998 by Diane Green and Mendel Rosenblum. Suddenly I got to take operating systems in college from Mendel and then Diane as we get too later in the story was running Google Cloud around the time I was leaving Google. So legends in this field and in this industry. And what VMware did is they made it possible to take a normal computer, right? Whether it was a server or PC for that matter, and run virtual machines on that computer. And why that's typically hard is a computer is usually made to run one operating system at a time. And that operating system is managing applications and making sure that the computer doesn't crash. Well, if all of a sudden you have multiple machines running on a computer at a time and one does something that you can consider kind of unsafe, right, that would stall out the machine or do something that they weren't supposed to do, that could break the whole model. But VMware's first product enabled an unchanged Windows and Linux run side-by-side on the same x86 box. This accelerated to even more interesting use cases. So you can actually hot swap VMs on machines at the same time. Let me give you a concrete example here. Let's say you're playing a game on a PC. It was as though all of a sudden in the snap of a half second that game moved to another machine midframe. So this was actually designed to be able to hot swap a virtual machine from one server to the next. So it didn't matter that you didn't actually have your own server because your virtual machine could float around as needed. So let's say that you have a hard drive crash. Well, you could have a snapshot running in the background and you could flip over to that one in real time. And so this notion of a virtual machine becomes the backbone for both the utilization point because one physical server can be used to actually host multiple virtual machines and a lot of the redundancy and fallback designs. And VMware wasn't the only one to do this. Eventually, there was the open source Zen project that is what Amazon and AWS first used. This kind of thread becomes better and better over time as all of the hyperscalers would figure out how to maximally virtualize everything that they do. And so now rather than me trying to run my pets.com and ensure that my product is getting to my customer and my website is doing everything. And then when it crashes, me having to stop everything and file the ticket and pause business to fix the crash, that's just abstracted out to the specialists to a business that is designed to solve this problem for me. And all I get is then continuous production, continuous service. And it gets cheaper for you every year. And it gets cheaper. Like the cost of S3 and EC2 has just gotten cheaper and cheaper and cheaper. It's an amazing business. And for many others that will get into the story, building these data centers and the utility business is not their high margin business. But for Amazon, this business has margins, which makes it a high margin business compared to their retail business. It's almost like if I were an airline spending all this time trying to figure out how to book a reservation and then you gave me a program that could do it for me, I can now focus on serving the customer and serving more customers faster. And it explodes. If you look at S3 in 2007, there are 10 billion items stored in S3 by 2009. That's about 64 billion. And by last year, 400 trillion items stored in S3. That's a really big number. It's a very big number. It's a lot of items. This powers the startup ecosystem in industry. Let's go back in time to that. So what was driving the growth of all these numbers? We're in hackathon city. We're having happy hours with engineers and folks with ideas coming together. And now you have an idea, you can drop a credit card down and you don't need to negotiate and buy hardware, no invoices, no contract, no sales calls. You're just you're up and running within hours. And AWS very brilliantly saw this as a pathway for short term and long term growth. In the short term, they can get a bunch of early stage startups using their compute. And that's not going to amount to a lot of money. But some of them are going to grow and they're going to be built on AWS. And so they actually had a business model of giving out free credits at these happy hours, at these hackathons to make AWS the default infrastructure platform to build a new company. And I think it feeds through from that go-to-market to their product design, right? Their product design was deeply unapairnated about what you were going to do. It was make it easy for you to get a server to go do what you want to do with it. Here's storage as simple as it can be. These are the simplest Lego blocks you can build on. And so simple. In fact, that then people built what feels like the same business on top of them. Ever heard of Dropbox? Dropbox is just an S3 application for this whole early period. It's storage. It's storage. Let's make it easy for syncing files from your computer to this new cloud thing and backing them up and syncing them and sharing them to other people. Dropbox and S3 are intimately linked. And so Dropbox is built to do exactly that. And it wasn't until 2015 that they're like, okay, we should probably look at the cost of this and they eventually moved off of AWS to their own servers because all they are is the storage layer. So that's a rare case where it made sense. That's all they are. They're still getting 1199 a month from me. That's right. Good luck in. So the Dropbox AWS story is a classic one, but there's probably no better partnership to exemplify this time than the one that Netflix had with AWS. Their ability to scale with these companies is really something to behold. And so in 2008, Netflix is still primarily a DVD by Mail Company for those of us that remember it. It launched a streaming service as a side feature. The leadership knew that there was something here. But in August of 2008, Netflix suffered a major database corruption in its primary data center for three days. It disrupted their DVD shipping, their streaming ability. It stopped their business. This was a huge wake up call. The recovery was very painful for Netflix. They realized that their on-prem vertically scaled systems were just too fragile and they couldn't recover quick enough from major failures. The leadership decided they They needed architecture. that was just designed to be more fault tolerant, designed to be elastic, designed to be globally available because they had aspirations of being able to stream their future business all over the world. And they concluded that they need to focus on their core business and building this out in-house was slow and costly. And so Netflix actually became AWS's first marquee all in public reference of a customer that scales. This proved to be vital. By 2015, Netflix was delivering billions of hours of content annually almost entirely over AWS and their own CDN. They were running thousands of EC2 instances and had all of their videos and the kind of canonical system of record in S3 of like the actual videos that we are watching. Now they did at some point realize that it was so important for them to own the latency and cost of that last mile of delivery, that edge. And this is a perfect example of a use case for CDNs. They launched in 2012 what they call open connect and the open connect appliances. This means that they would go into those meet me rooms that we had talked about before and drop a Netflix period box that would directly connect to your local ISP and they would do this for free. The ISP just says, "Hey, we have a lot of people trying to access Netflix. Let's make it better. Let's make it faster. Let's make it cheaper." It's a win-win. And so win-win and they cut out the CDN that they were using at the time. Netflix saves money, customers are happier. Everyone gets their videos faster, right? This way when someone on the ISP access is the newest popular movie, it's already close to them. Amazon doesn't actually even, they barely get hit in that moment. It's obviously running what's called the control plane for Netflix. - Yeah, 'cause if I have to wait three or four seconds for that preview to load, I might not watch that show. You might turn. And this is what powers streaming to this day. And the same model is what enables others to enable streaming at scale. All of this, this whole Netflix case study is the perfect flywheel and customer to show that AWS can scale with you and scale in a really challenging environment and be resilient and power global reach, right? Netflix was not just a US company at this point. - Yeah, for anyone that saw AWS and the cloud as a concept, as risky, as unable to scale, as not enterprise ready, Netflix helped debunk that for chief information officers around the world. And they would just do whatever it took. I can remember some story. I think it's from the acquired episode where Amazon would allow you to ask them to roll in a big truck, slurp up all your data into hard drives, then they'll bring it to their data center to plug in and dump all your data into your AWS instance, right? So they figured out what was needed to close the enterprise customers. And it's just amazing that someone who was known for selling books was able to so quickly build the brand around how to do this new private secure utility thing at scale. - So AWS continued to rethink what data centers are used for and how to build them out to serve the customer rather than have one mega facility per market. They introduced the concept of availability zones, which became clusters of independent data centers within a region, each on separate power lines, each on separate fiber paths, linked with millisecond private connections. And this availability zone concept and clustering was yet another piece of the data center revolution that AWS contributed to the ecosystem. - I feel like this notion of like US East and US West, this is how people think about their servers now, which is this abstracted notion of an availability zone. AWS continued to do this globally and this actually became a blueprint for hyperscale growth. To end the Amazon story here for now, I think it would not be wrong to credit them with really kicking off the utility scale cloud story and getting so many things right, giving developers the simple building blocks that they could use, going after cost, going after reliability redundancy, making sure to build the right guard rails for virtualization so that they could actually do this efficiently and build for the long term, using the same APIs internally so that it created a flywheel for them to move faster and bring those cost benefits back to the core business. It propelled them so far ahead in this business, they are still the leader to this day in the core cloud infrastructure product as a result. - You know, as we've been talking about the data center, it is simple at its core, right? It's a facility with storage, compute, and connectivity. And what AWS does, they take the data center and they enable it to build an economy on top of it. - To frame it another way, it used to be that if you wanted lights in your house, if you want electricity in your house to power those lights, you would put a dynamo to burn coal underneath your house, right? And if you wanted a website on the internet, you would buy a server and plug it into your home ISP. And that is insane to us today. What do you mean you're gonna power my house with electricity from underneath, ground my house, right? Instead, I'm going to hook up the shared infrastructure with all of my neighbors and we're gonna centralize our demand and build the cheapest, biggest infrastructure we can to generate the power. And that's the same thing happening here for the first time. This is the introduction of the concept of the cloud, right? People have had the internet where they're interacting with kind of publicly visible websites, private enterprises have servers, but the idea that you have your own storage over there and a Dropbox folder, or you as a developer could build whatever you wanted and access this floating blob of data that's to you invisible where specifically it is, but it's out there. That was the, I think, conceptual notion of the cloud. I've come to somewhat dislike the phrase because it feeds into the invisibility of the infrastructure. - Mm-hmm. - Right? It says like, "Uh, it is nowhere. Your photos are nowhere. Your storage is nowhere." And that's so far from the truth. It is somewhere. It is in multiple locations, in fact. The cloud is almost the designed antithesis of the glass room. The glass room where you wanna show off, this is where it is. This is where the compute is happening. It is only happening here. It's happening right now, but it served a good purpose in explaining this notion of migrating to this unknown location and has remained pretty sticky, I think. So while Amazon was out there selling books, a little startup out of Stanford was helping you organize and access the information on the growing worldwide web. - Google was founded in 1998, and shortly after it was handling about 10,000 search queries per day. By the end of 2006, it was processing the same number of searches every second. Google acquired YouTube in 2006, a year after its founding. And at the time, it was already one of the fastest growing websites in the world with 100 million video views per day. - And I remember folks having their first web mail-based accounts, like a hot mail account, they'd have two to four megabytes. Well, Google notoriously on April 1, 2004, announced and launched Gmail as an invite-only service that had a full gigabyte of storage. - I remember you could get access if you referred in. - If you referred in or they started going on eBay for like $150, there was a hot ticket. By 2010, Gmail had over 150 million users, and of course with more gigabytes of storage available. - It was amazing. It was from the very beginning, the front door to the internet, but it was also going to need to be an infrastructure company. - And so you go back to 1999, year into Google's life, and an engineer named Erz is being shown around as part of his recruitment by Larry Page, the CEO and founder. Erz says you couldn't really set foot in the first Google cage because it was so tiny. The cage was seven feet by four feet with 30 PCs arranged on the shelves, providing the world with more Google than it could handle. Our direct neighbor was eBay. A bit further away was a giant cage housing deck machines at Alta Vista. All of this was hosted at Exodus in Santa Clara, one of those co-locations that we talked about earlier. - So Google competing with Alta Vista and all those others at the time, it was powering part of Stanford's search. - That's right. - Running off of 30 PCs. - Running off of 30 PCs in this co-locent center, it cost Google about 1400 per month, per megabit per second of data. And so they had to purchase two megabits per second at the time. One megabit was about a million queries per day. From the beginning, Google looked at its servers and infrastructure differently. It was never interested in taking the tried and true path of buying the sun or even HP boxes and using those. The most memorable stories of this time that explains this is the infamous corkboard. So in the early days, Google engineers literally mounted motherboards on corkboard. They had $15 box fans pushing air across. They had zip ties holding it together. And in a traditional IT philosophy, this is heresy. But the idea was that if a part doesn't add reliability at fleet scale, strip it away. - This all set in motion, this ethos of questioning the assumptions and ultimately deep, deep vertical integration of their infrastructure. Some might really argue that that was like Google's superpower and still remains hope to this day. The Apple's piling all of that thinking into making the perfect iPhone, Google's doing it to their data centers. - And so Google had a number of innovations that came from this early time. One was around how to do power backup. So the standard was to spend for a facility-wide, big battery. We'll buy 2000 Google's questioning this model and seeing the waste. in the large battery. And so they actually put little batteries on each server. Again, they were accepting that one might go down, but that was okay as long as the whole fleet was reliable. And so the key to all of this working was moving reliability up the stack into the software layer. Google was not using off-the-shelf file systems and off-the-shelf software. They built everything here themselves. They assumed that a hard drive should break. And so they built the Google file system. This was like 2001-2002. So this is a distributed file system. They would take files, chunk them up, and spread them across three different servers, no matter what. So you could always have resiliency. And also, this would make accessing the search index much, much faster. They built in 2003, Borg, cluster manager, which ultimately was all about managing where jobs were running on which servers and which machines. And really was like an extension of the virtualization and predates Kubernetes, which we'll talk about a little bit later. So these were all the software components, but they were also questioning many of the physical constraints that people had liked to have with servers. And so most notably, they did some of the first experiments in really this hot-Ile, cold-Ile, air flow containment. Traditionally, data centers were just a bunch of hot computers in a room, and they would just blast cold air into the room. So you'd have to keep the room as cold as possible, often uncomfortably cold. And there was no real thought about getting the hot air out of the room. Over time, it became accepted wisdom that you should point servers in one direction, get the hot air out of the room, encourage that to happen. Google realized that the more you took that to an extreme, the better. So they would put sheeting up and isolate the hot-Ile from the cold-Ile. Because then you could actually extract the hot air more efficiently and not mix it back into the cold. And so what we mean by cold-Ile and hot-Ile is that cold air needs to blow across those exposed chips and rather boards that would extract the heat off of the server into the hot-Ile. And then the hot air from the hot-Ile would need to be extracted out of the room. And so Google became really a leader in designing the best airflow there. Which is a concept that continued for years. You also think about the traditional data center and this idea of assuming something's going to fail. And you build around that. Whereas in a co-lo facility or when you're renting out these cages, every server had to work because it was a different company utilizing it. And not only that 20 years ago, you had specialty cleaning crews that were moving around the data center. And they would sweep up the room and then analyze the contaminants. What they had just sweeped up to understand how to continue to optimize the cleanliness and space around this. The idea of laying in a corkboard and a zip tie around it and assuming something would fail, it really takes the data center concept that existed today and completely flips it on its head. And then as we'll see drives better performance. All of this comes together when they build their own real first scaled homegrown data center. I love this idea of don't think of a data center the way we've been defying it as a room pack with servers and power and connectivity. Think of it as a warehouse scale computer. Errors wanted us to think of it not as many machines, but as one machine. So you think about the components of computer, right? The storage, the memory, the compute. And Errors saw what they were building out over time in their data centers and realized that they were just building a large computer that just happened to be the shape of the warehouse. Yeah. And if you accept this premise, then you stop trying to make each server perfect. But you start thinking about making the fleet reliable. And you design entirely differently. And so this is about 2004. You have a wild man named Chris Saka. So before he was a famed investor, he was apparently a young, sloppily dressed individual walking around rural Oregon, looking for shuffle ready enterprise zones where he could find some tax breaks. And he's walking around asking for such astronomical quantities of power that allegedly a nearby town suspected him as a terrorist and called the Department of Homeland Security. But this is just what he was looking for. And Dalles, Oregon had a site for him, 30 acres next to a decommissioned aluminum smelter that once drew enough power to power the needs of a small city. And Saka was ecstatic. He said it was visionary. This little town with no tax revenues had figured out that if you want to transform an economy from manufacturing to information, you've got to pull fiber. And so Google went on to build out Dalles, Oregon and bring all of their innovation to bear to increase the performance of the status center. And so what that meant was low cost steady hydro power from the Columbia River. The Bonville corridor providing high voltage transmission. A cool dry climate that lets you run a cooling process. Most of the year that's very economical. Long haul fiber that traces the river's edge. And a town that was hungry for a new tenant. This all came together to launch Google's first warehouse scale computer to drive down unit economics, improve the performance of a data center. And therefore the unit economics of the entire fleet. And I would argue I think this is potentially the first built from the ground up hyper scale data center. They're building to solve their problems. But their problems are at this point scaling so rapidly that they're thinking about how to optimize all these things in a way tuned with software, tuned with virtualization and all these things that this is the new blueprint for the future that is still how we're generally designing data centers today. They had a head start as thinking about infrastructure from the get go making it part of their DNA. And it's not as if the other companies were standing still. They were building concurrently. This was a race and they had to do it quietly. And Google was the first to get operational in 2006. They quickly replicated this elsewhere in North Carolina and then in Finland testing different climates to optimize using seawater, using hot-out containment, other ways to optimize cooling and performance. There's others that kept pushing on, right? Like the power supply system. So normally each server you think about plugging into the wall, it's converting AC to DC and then stepping down the DC for all the various components. Google instead realized that if they could just have a higher voltage DC current coming into the rack, they could then convert once cleanly and do it late as possible. And so that higher voltage, you'd have less loss because it's a higher voltage and you get efficiency gains across the whole path. And so at the scale that Google's building servers, they can optimize every single part of the stack. This 48 volt DC they later announced in 2016 publicly with double digit efficiency gains that then the world could incorporate into data center design. Now we've talked quite a bit about performance and efficiency and there's good reason for that because a lot of data center companies were hemorrhaging money on power. It was enough impact on the bottom line that there needed to be focus on this. We had the distinct pleasure of talking to one of the OGs in Data Center's Christian Belady. And he told us this fantastic story of bringing the metric for power efficiency to the industry. It's called PUE. PUE or the power usage effectiveness. This is a simple ratio of the total energy consumed by a data center divided by the actual IT equipment that you're trying to power. So it very easily spits out what is the overhead and running your data center if it's two. That means that you have twice as much energy used to the energy that's going to the actual servers. And how did this metric come to be such a simple beautiful metric? So Christian Belady was working at HP in the late 90s and HP had a customer in Japan called NTT Dockamo. And Christian had been in the industry for so long that he had formulated enough standards through HP and through the industry to come up with 10 best practices. And he took these 10 best practices to Dockamo. And they were like, yes, this is great. We're going to implement all this. It's like fantastic. Come back in three months and we'll do our review. So he goes back to Japan and they had printed out everything and had big stacks of paper on the desk and everyone's in a suit and tie and a hot room. And they come to Mr. Belady. We've done everything you said. Here's all of our reports and nothing's changed. The servers are all still the same. It's even hotter and hot aisle and no one wants to go back there. And so it doesn't seem like it's any better. That's right. We're still hot when we go in there. We don't think this is working. We're going to go back to the way we were doing it. And it drives Christian nuts. This is the whole point. It was that you actually want the hot aisle to be hotter and the cooler aisle to be cooler. That separation is what drives the efficiency. You don't want the air to mix. So he knows in his bones. He's like, no, you're running in a more efficient way. You have to see that. Prior to this, NTT couldn't see the change or measure the efficiency gains of the hot aisle, cool aisle containment or the other recommendations that Christian was making. And so he invented this incredibly elegant metric and it kept it internal to HP. They continued to implement it in HP for six years until his good friend, Chris Malone, suggested he publish a paper and present it at at Kenbrill's office. uptime institute in 2006. At the uptime institute, they present the paper and they kick off a new organization called the Green Grid to publish PUE and other metrics. It was Christian, Blady, Paul Perez, Bruce Shaw, Larry Vartle. These folks later became the CTO of Dell, big teams of AMD. And they thought it was just going to be an internal kind of, here's the thing we tried. It hit the industry like a storm. Because now the PUE race was on and Google wanted to win the race. Many, many enterprises had numbers around two. As in twice as much power went to the lights and the cooling and everything else to the amount of power that actually ran the IT hardware, Google pushed it down to 1.1, meaning only 10% of the power was not used to power the compute. Their internal teams took hold of this and ran with it and they used this rethinking of the data center from the ground up as one computer system, as software being the reliability layer, not the hardware, and innovative geographic placement to drive performance and drive PUE and it's come to a point where it essentially can't be optimized past one. Before we get to a far ahead of ourselves, let's talk about what it is to go to one of these data center campuses. You're driving up into vast open space where you suddenly see massive buildings growing out of the ground. From the sky, it maybe looks like a distribution center, you can see some steam flowing out from the building, but you drive into this campus as it is. You start noticing why would there be so much power infrastructure? If this is just for a warehouse, because you're seeing big battery packs or extra turbines or generators. The other thing you'll notice is security is everywhere. You have to pass through a series of checkpoints as you get to the front desk. Maybe they're powering Netflix, but they're also powering Pentagon operations, so they have an enterprise obligation. So security is paramount. So you're approved, you manage to get through multiple layers of gates. So there's a moment, the moment of big reveal, when you walk in, and it finally hits you, the rows and rows and rows and rows of machines that just go on further than the I can see. These facilities, they're bigger than a football field, and they're lined with these racks of blinking lights and machines. You're hearing this constant roar and you're noticing the temperature where you walk in. It might be fairly comfortable. I think 80 degrees is average in a Google data center today. What did it used to be like? I had to keep these machines cool, and the best way to do that was to chill the room. So it was frigid. The entire room was blasting with AC. Back in the 80s, you just wear that thick sweater all the time, and now they get to comfortably hover around 80. And that's because of that really good containment that they did of hot and cold aisles. Now you've taken in the machines, you've adjusted the noise. You've given your earplugs to put in to get comfortable, and you look up and you start to see the infrastructure that's coming in and powering these servers and connecting them. Google had famously bright colored wires and marked where the plumbing is, but it wasn't always like this, right? If you go back to the 80s, I think it was quite different. You now are building right on the floor, but one of the ways to cool the data center decades ago was to raise the platform. So you actually would look down and you'd see that you're walking on panels above a hidden space where cold air is being pushed through. Per the trend of everything we've talked about with Google driving the evolution, what is the cheapest way to most efficiently do the thing we're doing and it stopped being those raised floors over time? Hopefully that gives you a little taste of what it's like to walk into one of these. There is a nice Google podcast called Where the Internet Lives that really give a really nice tour of exactly this. They bring a good Google and latitude media production to it where you can actually hear the sounds as they record inside the data center. So the worrying is very palpable. Google's investments continue to grow. They continue to vertically integrate deeper and deeper. They grow their insights. They start buying up their own fiber. They took advantage of some of that dark fiber that we mentioned earlier. It snapped me a lot of it up. And all of this is a capacity strategy. Ultimately, they want to have the capacity redundancy to drive their own cost down. While mind you, they're building the highest margin best business in the world in the form of search and ads. So that whole flywheel is allowing them to continue investing in this infrastructure. Google had this explosive demand as they are the web company here. They have explosive internal use driving the value of the vertical integration. But the whole industry sees what's going on with AWS after it's launched. And Google's one of many companies is that they realize is they have some assets to put towards that too. And so they put together a team that launches App Engine in 2008. So that's you run apps on Google's infrastructure, but was very opinionated about the app. It only supported Python with a particular framework. So it's quite different than Amazon's basic building block EC2 approach. And it wasn't until 2013 that Google launched a general purpose virtual machine capability. As a result of this approach, as well as Google's lack of enterprise sales and marketing motion, it took quite a while for their cloud program to get running. Things started to shift in 2014, 2015. Google gets public about its containerization strategy. If you recall our discussion of virtualization, well, you don't actually need to package up the whole operating system. You don't really care that it's running windows or Linux. You actually generally just want to run the application. Containers are really a way to do that at the application level. You see this continue even further more recently with the idea of serverless functions. And so Google in 2014, 2015 took the world by storm by open sourcing Kubernetes and the Google Kubernetes engine. It very quickly became the default way for programmers to use containers and migrate away from virtual machines. Now they still didn't have an enterprise sales partnerships muscle, but Ben, you were there at that time, right? That's right. And so that's when they brought in Diane Green, who we previously talked about was the founder of VMware to help build that muscle. So by 2018, their cloud share was 7%, AWS was 34% and Azure was 15. So they were a distant third. And through to 2019, a new leader came in who's been the leader since and he has made big pushes in his growing mass. Their cloud business is now $13 billion business growing 32%. Netnet you look at Google's story and it's interesting. They've always been technically ahead, especially internally. And they've always had amazing dog feeders, right? We need people inside using their infrastructure. The problem was they did not know how to sell to this customer. They were not really a developer platform, especially to the enterprise. And this is a market where Amazon had cracked that. They really had to learn new skills despite having the best tech in the entire ecosystem. I think the other story is that Google's vertical integration sometimes went too far. They figured out the best way to do this for themselves, but sometimes that led to blind spots in what someone not in the Google ecosystem might need. So the idea that every developer is going to build a new Python framework that Google releases is a little self-serving when turns out engineers might just want maximum flexibility. Or she Amazon deeply understood. And it all starts with Erz giving the industry an entirely new way of thinking about it. Treat the data center as a single evolving machine. Build the hardware to suit the software that will survive its own failures. Put the buildings in thermodynamic, optimal and grid optimal locations and publish your math, showcase your performance and your PUE so that everyone can have FOMO and try and play catch up. So Google was spending billions on land, buildings, hardware, designing its own infrastructure and overturning the data center world by building software to bind it all, which forced the entire industry to meet hyperscalers like Google on its own terms. Competition was heating up. It wasn't just Amazon and Google. There were even more household names, quietly transforming themselves. So by the mid 2000s, Microsoft was unquestionably the software king. Windows was powering 90% of desktops. Office was licensed to print money. But things were changing. Broadband penetration was climbing. We started to have the emergence of connected phones, BBM and blackberries and things like that at the time. And internet native companies, they saw Google and Amazon building and climbing and proving that you didn't need to buy software in a CD to run interesting computing applications. Remember the cloud, as we've been talking about, isn't just about backend developers, applications themselves were starting to move not just their backend, but the front end to the browser as well. Right on the browser. Remember, Salesforce was leading the charge here. They launched with this idea of no software all in the browser and by 2005, they're doing nearly $200 million in revenue. And capturing the mind share, proving that this could be the future, not just of consumer fun applications, but also a business product like a CRM. Enterprise, B2B software. And so you're there saying that Microsoft, like that is your greatest fear that you're not going to be part of the application and operating system of the future. That is your whole business. Microsoft at this point, it's like they didn't run any web services 10 years prior gates had sent his memo of the internet title wave. They had invested, they bought and scaled hotmail. They had MSN. They had Xbox Live. But these were all these self-contained services. They hadn't yet thought about how to expose an infrastructure to others. to Bilban. They loved building operating systems that would ultimately run others infrastructure. They had built one of the best businesses in history doing this. They write this code once, and they print as many CDs as they can, and this amazing, amazing, high margin business. They didn't have to go buy other people's hardware. They print the CDs, they print the money, and this idea of infrastructure as a service was radical. Inside of Redmond, Washington, the pivot for Microsoft is framed as nothing less than existential. So, as is often the case in a company motion like this, it's about people and talent. And so, they bought a company, Groove Networks, that brought in the famous Ray Ozzy, who had previously led the development of Lotus Notes, and brought him into Microsoft and put him as a co-CTO of the company. This was so existential important. I think about what Meta's doing today to staff up talent. It was that of the time. The cloud is the existential thing. We need the leader of the cloud, Ray Ozzy, to come in and lead the way through this charge. If we're going to bet everything we've built on this pivot, we're going to bring in the best talent. And so, they buy the company, they put him in charge, and this is this 5,000-word manifesto about what the future of Microsoft needs to be here. And so, it's worth reading the whole thing, and we'll link to it in the show notes. But, and I, would you read that key section that really stood out? And so, in Ray's memo, he says, "Computing and communications technologies have dramatically and progressively improved to enable the viability of a services-based model. The ubiquity of broadband and wireless networking has changed the nature of how people interact, and they're increasingly drawn towards a simplicity of services, and service-enabled software that just works." And then he ends that thing. Businesses are increasingly considering what services-based economics of scale might do to help them reduce infrastructure costs or deploy solutions as needed and on a subscription basis. To just give people the end result that they want and demystify everything's happening. That's right. And he ends with, "We must respond quickly and decisively for a company of Microsoft's scale and history." That's really trying to be a wake-up call. And didn't Gates and Bomber kind of give Ray Azia blank slate? Like, "You're the leader here, help us understand the way the future is." Not just that, they let him carve out and run this separate from Microsoft Server and Tools Business, which had built Windows Server, had built SQL Server. Would have been the obvious place to try these things, but they knew that this was bigger than an evolution from what they were doing in their previous software business. This was a transformation of fundamentally different product and different business model and different go-to-market motion. So when everything has to change like that, it is very hard for the leader of the kind of incumbent thing to keep going. Let's make this concrete. Like, what does this mean for a server or a data center? Microsoft, again, had Windows Server, had SQL Server. When we talked about those server rooms that a company would run, they would very often be running Microsoft software. This is this IT sale that they had perfected by this point. You would sell the Windows Server license, it would be active directory, it would be running an exchange, an outlook for your email, it would be running your calendar, and all of these things inside your company's network. It'd be running Windows on their PCs. And so this is beautifully connected thing. And this is saying, no, no, no, we've got to move that server from your bottom line cat-backs as a company into op-ex, into our data centers. And this is a transformational move from what you're providing at the end of the day, from just software that's going to run on someone else's computer to actual services. Ray Ozzy goes for it. He's Daphs of a team. They call the project Red Dog, and they bring Dave Cutler, who had led Windows NT prior and a legendary programmer, in to help lead the architecture of this new project. And so they're putting together this cloud strategy that is revamping Microsoft internally. And at the same time, ramping up a direct competition to the front door of the internet, Google. As they evolved an old MSN search product into what then became live search, and eventually being, as we know it, in 2009. And this was a moment that kind of rippled across the industry, especially for Google's dominance. I was at Microsoft at this time. Oh yeah. I joined Microsoft out of college 2008 and was there. In this moment of they had Windows Live search, they was briefly called it live search. Then this big rebrander bang. And Microsoft was willing to spend a lot on this repositioning on this marketing so much so that they gave Yahoo a very sweet deal to power Yahoo's search and ads and gain some infrastructure market share. This catapulted them into getting up to 20% share, which is no small feat in an exploding market that Google has been dominating. It's important to remember that like doing search well and search ads well is a really hard scale problem that essentially only one other company had cracked. And that was Google to respond quickly with this active updated index of the internet and do this auction for the right ad unit and all of these things that you have to build is what propelled Google to build their vertically integrated servers. And Microsoft was now putting themselves in the position to have to solve those seem really hard technical problems. In addition to those really hard technical problems, we saw with Google that in order to effectively run this search business, you have to have a lot of compute power behind you. You now have to get into the data center business and build out infrastructure that is your own to drive down the cost, drive up performance. We mentioned Christian Belady previously. He was at HP in 2007. It's a recruiting call from Microsoft. And his initial reaction is what the hell's a mechanical engineer going to do at Microsoft because Microsoft is known as the software company. And so he turns them down a couple times. After the third call, he decides to go up for an interview, enjoys the process, gets an offer, but doesn't think much of it. And I believe he's at his parents house when he suddenly gets an email from Bill G Bill G at Microsoft.com explaining why he should accept the job offer. And Bill essentially says everything's going to the cloud. We're investing for the cloud business. And we need to build out this new infrastructure as core to the future of Microsoft. This seems like a wacky idea to go there. But Bill G, emailing me like you're there with your parents and got to give this a go. Let's pack our things, move to the Seattle area and join Microsoft. And so at this point, he starts there and Microsoft had just finished building their first real more integrated data center build out. This is in Quincy, Washington, Eastern Washington, 13 megawatt build. And he walks in there and everyone's looking around. And the first thought they have is we're going to get fired because there's no way we're going to fill this thing with servers. There's just no way that we have the demand. And this was their first massive scale purpose built cloud data center campus. This was Microsoft entering the infrastructure business. And similar to what we saw with Google at the Dallas, a lot of the same things that attracted them to Quincy were power from the same Columbia River. They're getting cheap 1.9 cent per kilowatt hour power at the time to power this. And the climate was that dry, cool climate that allowed them to be really economical with using outside air cooling for much of the year. Then multiple fiber routes. It's in Washington right so they can connect their Redmond headquarters directly to Quincy, the environment, the fiber, the power. It all comes together and they like Google knew they had to work with the local community and the local government. And so they actually worked with the city of Quincy to build a Quincy water reuse and treat and recirculate the cooling water to reduce the dependence on the local community. They built Quincy. They haven't been running. He's worried they're not going to fill it. But they kept going. Azure ends up launching 2008, which we're talking about in a moment. But I think it's very easy for a big company that's bett a lot of resources building something internally to think that they have an idea of what demand is going to look like. But they overbuilt in Chicago. Then they had it mothballed. They tried to sell it. And then nine months later, they were so happy they didn't sell it because they needed the space. There's layers they tried to sell it to the government. So what's going on there and what's what's happening is they're launching this project red dog, which would become known as Microsoft Azure. And it's hard to predict the scale of the utilization of infrastructure as you launch it. So let's go back to October 2008. The big reveal. This is the Microsoft's big annual professional developers conference and Microsoft unveils not Microsoft Azure at the time, but Windows Azure. And this wasn't just Windows running somewhere else. It's a platform for developers to build and host applications on Microsoft infrastructure, paying only for what they used rent some storage and compute pay for what you use. And the key thing that you mentioned is designed for those Windows developers. So it used all the familiar tools, you know, the dot net framework and visual studio and all these things. And it made it easy for you to take your work that you've done over there that would run on Windows server, right inside of your office and take that same code and shift it to run on Microsoft servers. What a powerful integration and move. It kind of split the thread between AWS, which gave these really basic building blocks as three and EC two of this unapologated like run your own operating system, bring your own full thing and we'll just make it work. They were agnostic to what you were doing on top of it and how you're doing it. Exactly. To Google App Engine, which was overly specific and it's cute about it in a way Microsoft was almost more Google's approach, but it was the same code and the same thing that a developer had already used. And so they had this massive developer ecosystem and integrators and this whole kind of motion around it. And it wasn't just easy for the developer community. This is Microsoft that has an enterprise sales engine. They've got to go to market They have the finance department that knows how the sales team is going to operate and how to budget for it. And so they're bringing this world-class distribution and most importantly, the trust from this massive customer base to bear with Azure. And the adoption is extremely rapid because people are already embedded in the ecosystem. And here you are coming with an improved product and the enterprise base adopts it very seamlessly. Yeah, and it's not the story that Amazon has to make for the first time, which is, hey, come build your thing with us in the cloud and introduce this. This is four or five years later. And IT say they're like, how do we think about the cloud? We don't have an agreement with Amazon. That's not a vendor versus like Microsoft. You're like, this is the agreement we have. This is the IT relationship we have. This is the full trust. If you're telling us this is how we can move to the cloud in a seamless, secure, safe way that it's going to integrate with Active Directory and Outlook. And it's just, I don't have to run the machines in my closet anymore. Sign me up. Ironically, I think some of the people that had the hardest time signing up were the engineers and product managers in Redmond, Washington. We were actually talking with Ben Gilbert required about this. And he at the time, this is 2012, 2013, was working on the web version of Microsoft Word. So Microsoft is finally going to compete head to head with Google Docs on product and ship a version of word that you can use via your browser. And of course, the desire was for everyone to use Azure today. Absolutely not. They said, no, this is the custom server we need. We need this many of them can scale this big. And we had to do some custom JavaScript rendering, all this stuff. We want our own servers, probably still in Quincy. But we're not using the Azure API and layers. So the own employees are demanding all this heterogeneity and resulting in a wide variety of server ask a use. Amazon for context, building out AWS, I think that's something like 20 different types of machines that were running AWS. Google in their whole build out standardized on this vertically integrated commodity built server design. And so they would just keep it down to a handful of machine types. So supposedly at Microsoft, there were dozens and dozens of different skews that Microsoft was maintaining. So instead of calling this a server farm, someone call it a server zoo. You just think about where they're coming from in the evolution of all these different web services and different teams building their own thing. It's a whole different world. And around this time, I think energy starts to become an interesting layer as well. So similar to Christian getting recruited in 2007 thinking, what the hell am I going to do at Microsoft? He then goes out to recruit an energy expert Brian Janis in 2011. Microsoft saw the need for an energy person to be in house. I say Microsoft saw the need, but I think Brian was confused. Brian was like, I do energy. You don't need an energy person. And his thought was this is a dead end job. Being an energy person at the software company doesn't make sense because though what they're trying to do is all about land, it's all about fiber. Energy is a distant third or lower down the list in the priority of how you think about scaling this type of infrastructure. Yeah, I think at this time for context utilities were happy you'd show up with your 10 you know, Quincy 13 megawatt data center and you told you're like sure great sign up. Yeah, generally power demand is relatively flat. There's excess power available and they're happy to sign you up and increase their profits because it's not going to drive any new infrastructure needs on their end yet. And how things change this is 20 2011 2012 fast forward a little more than a decade where we are now and we'll get there. But Brian took the job and end up building a phenomenal team within a few years his team was actually the decision maker of where we're going to build out Microsoft infrastructure. And so it's a lead into we actually start to care about where energy is coming from and the whole complexity around where you're sourcing and doing these clean energy buildouts that Brian's team was leading the team thought is a real kind of planning and challenge right and so they're bringing this up to leadership around hey, we need a lot of money to invest now to build this out. They're doing their annual budgets they're projecting years in advance they're getting into their mid year review Microsoft in this weird transitional phase they have been that they're still growing they have other services Xbox is booming as you're still early in in so it's not the dominant use of their internal server build out. But it's a lot of different teams to juggle and predict and so it like this forecasting challenge is a real one you bring this to leadership and what are you there to say. And you've talked to all the business groups to figure out what they're thinking and you go to bomber if you got your prepared notes and within a few sentences you get interrupted and Steve bombers like the business groups don't know what they need. Let me tell you how to do this. And he says give me the excel sheet you just draw a straight line from how growth has been going last few years and you just project it out and you just build the data centers that that line shows you to do. Just a straight line from here to 2020 the business groups they don't know what the plan is there's actually a lot of wisdom to what bombers saying because you realize that every team is trying to be excited about what they're building and Azure is going to have all this demand but no one really knows especially more than 12 maybe 18 months out. So how can you plan if you're building long term multi year build out for what anyone's going to need. In this moment of growth it was quite difficult and the Janus team found that he was consistently under forecasting but he did go back many many years later and he's kind of curious how accurate where his forecast compared to the straight line in 2020 he looked back and bomber was off by how much 5% 5% so they should have just followed the line. So sure enough Microsoft keeps at it and the power team at Microsoft the data center team at Microsoft becomes one of the biggest builders in this industry they shift from leasing square footages to megawatt based deals as they become a major major hyper scalar. There's a big learning journey for them right because they were a software company and they had to grow very quickly into an infrastructure company it was not an easy pivot to make and they had to do that globally the Microsoft team had the blessing of this entrenched enterprise customer base that was very loyal very profitable and very global. And so their go to market strategy was actually to follow their customers and that meant that within a few years of launching Azure they were getting calls from customers in Frankfurt and Amsterdam in Singapore if they're German monthly national had data sovereignty questions Microsoft would go and stand up capacity in the region. So by 2015 they had announced over 20 regions which is actually more than Amazon at this time each anchored on an interconnection rich metro and typically supported by entering with co location facilities perhaps the digital realities of the world before moving on to dedicated Microsoft campuses once they built up enough critical demand but this go to market strategy forced them to innovate and expand all over the globe and it drove. Massive results quite quickly and to that point fast forward today and Microsoft's business. Is in the significant part the Azure business Azure is a core growth reverts now 75 billion dollars a revenue a year in growing 40% year over here and so this company that was the software company we write software we sell software those are core platforms is our products that's what we do. Is no longer just a software company they are in large part a data center company. So we've gone through EWS Google and GCP Microsoft and Azure but there is another company that many thought for many many years was just a toy let's talk about Facebook. Facebook in February 2004 Facebook running out of a single server in Zuckerberg's dorm room in Harvard University early days caused crashes in fact many of his earlier products including face mash the hot or not for Harvard campus melt down the servers in the Kirkland House and the IT department at Harvard was not pleased. But then it becomes real company right they start expanding to other campuses and you've got different schools with putting on different servers different data centers in a way that I think creates like automatic resiliency because you can't if the server crashed for whatever pandas not going to bring down Harvard. This isn't their concern right there they're building this viral kind of game changing platform that is exploding probably beyond their expectations and new campuses are signing up so they're not really concerned about the infrastructure but it's just growing organically by the time they launched newsfeed you've got hundreds of millions of people refreshing this site uploading photos by the billions opening up messenger just as a reflex of the thumb. Somebody spain all this money they raise VC dollars and they're not happy spain all this money on fancy servers and co-location. And so they're realizing that why is everyone making so much money on us there's got to be a better way to this and there's two broad ways that they go about this one is similar to what we've seen with Amazon Google and Microsoft which is the thought that hey we can do this better cheaper if we build our own custom data center. And so their first purpose bit facility was announced in January 10 and began operations in 2011 in primeville organ and they soon. You didn't follow it with North Carolina, Sweden, Iowa, and expanded rapidly. But Primeville was just like all the others, quite intentional and quite strategic. Yeah. So similarly, in this northwest region, you had cheap power. You had a dry, cool climate. And you had organs, incentives just trying to come into play where they offered long property tax abatements and many improvements they made. All of this combined to make a great site for their first lighthouse data center. And if you look at the numbers of server growth, if you go back just to 2008, they had 10,000 servers, 2009, 30,000 servers, supposedly by 2010 when they are really kicking off and building the Primeville project, they have around 60,000 servers that they're trying to figure out how to scale out. And so they decided to engineer the building and the servers together. This is like very purpose built facility. And just like the others, they realized that they can beat industry standards. And so they very famously and publicly came out with this first purpose built facility with a very aggressive target of a 1.15 PUE industry averages, 1.5 historically, they were north of two. And they were able to report 38% less energy, 25% lower cost against prior facilities. And do you think they ended up beating their PUE? No way. Smoked it. 1.07. So that means that only 7% of the energy going to run the data center went to any thing other than powering the IT equipment. That is phenomenal. Amazing. Did they keep this to themselves? How they did this? I think this is what sets Facebook's approach apart. This first part they needed to do and they executed extremely well. But the second thing they did was far more revolutionary in a very secretive world of data center development, where each hyper-scaler kept their builds themselves. In April of 2011, Facebook opened source their blueprints. They announced the OpenCombu project. It's hard to overstate how radically different of an approach this was. This is an industry that kept the design of the servers in the data centers extremely secretively. Like they viewed that as this core IP and differentiation and notions of security. And Google would publish papers, especially on things like PUE and be visible about metrics that they wanted to highlight, but they famously didn't let anyone into their data centers until this really changed the game. Pretty interesting, their motivation. They look around at all the other big data center companies and they're seeing all of the margins that are being made. And they're like, "Well, wait a minute. If we publish our blueprints and we get everyone else to buy in and publish theirs, that means we can drive down the cost by standardizing what we're building." Ultimately, what they want is their suppliers to be an increased competition. And the best way to do that is not to make one deal with one supplier. But to say, "Hey, suppliers, this is what we need. This is what we like. You make this. We'll buy it." As long as the cheapest one out there. Yeah, it flips the power dynamic. It now, instead of the vendors and suppliers dictating what the specs are and having multiple different specs for multiple different customers, they're able to standardize it and say, you're going to respond to the OpenCombu project standards. And it was directionally aligned with all the innovation that we saw at Google. There's actually a big parallel echo here, right? Google built all of these in-house software orchestration tools that we talked about that made the data center reliable. And Facebook grew up in that open-source world and benefited from that and was able to build on top of that. And so it's not surprising that they're the ones to look at the hardware design and say, why is this any different? We don't want to be the custom-only developers of this part of our infrastructure. It actually benefits us to have other developers and the ecosystem. In part because, and this really gets to a business model motivation, we're not trying to sell that innovation. We're not trying to sell anything to another business other than ads on our feed. The better and the cheaper this infrastructure is, the better our business margins are going to be. Yeah, we want to take this cost item and reduce it. And so we can put more of our budget, more of our focus onto the core business. We're looking at, and they're like, well, the industry is using a 19-inch rack because that's what the telcos did. Like, well, that's not suiting what we need. There's no reason for the 19-inch rack. It was just a legacy element. So Facebook decides to do differently. They widen it to 21 inches and publish that to make it more efficient and fit their needs better. This is funny thing how companies are open strategically. And so it's important to be clear-eyed about what they're open about and what they're going to do. You know, would Facebook publish publicly the exact way that their newsfeed ranking works? No. But that's very different than this infrastructure layer that benefits them when the ecosystem adopts it. I think one thing that's amazing about it is how much the industry rallied around it and started to join it. OCP ripples across the industry. Microsoft starts bringing their designs into it. Google contributes big improvements like we talked about their 48-volt design and telcos and everyone joins this and it has the intended desire in the flywheel effect. It did normalize the idea of transparency. And this is right around the time where companies like Google at first were publishing their PUE. And so this notion of we can be transparent with our results as a way to drive the industry forward, bring down costs and make this whole part of the business easier for everyone and spur more competition. This kicked off the sustainability race amongst these four hyperscalers. I was able to start all the way back in 2007 when Google announced that they're going to have an operational carbon neutrality. Each of the companies followed and they were serious, right? Microsoft commits to carbon neutral operations in 2012 and it drove site selection. It drove their power purchase agreements. It drove how they negotiated and determined who they were going to work with and their relationship, their utilities. And it just accelerates and accelerates between Amazon. They kick off the climate pledge to get 9-0 by 2040. You have zero waste goals. You have water positive by 2030 by Microsoft announced in 2020. Google catches up to have a similar announcement at the same day a year later. So you just have this era of accelerating commitments and goals. And I think it's good to ask, you know, why is this? Is it just feel good? And like having worked at some of these companies, I do genuinely think the leadership wants the company to have positive environmental impact. And it is the economically right thing to do as well. These goals of efficiency lead to your data centers, which are a growing item of operational scale, costing you less and less. The other thing this does is catalyzes different renewable buying and procurement habits, right? Where these teams, like the one led by Brian Janus at Microsoft, become these energy buying procurement machines. They're looking to help accelerate that next solar project, that next wind project, to then be able to buy via what's called a virtual power purchase agreement by the kind of accounting for the clean electrons. Google pushes this even one step further to say, hey, we want to buy electrons that are generated at the same time that our data centers using electrons. So we want this notion of 24/7 carbon free electricity. And so this really pushes the clean energy ecosystem forward, I think in a material way. And if we look at these three commercial cloud businesses of AWS, GCP, and Azure, they are behemoths. As we said, Azure hit 75 billion last year. GCP surpassed 50 billion. And AWS was on a run rate of $111 billion for that business. You total that up. That's $236 billion a year, spent only on the direct cloud infrastructure businesses. And let's put this in perspective. It's $236 billion across these three companies spent on the infrastructure. US consumers spend $500 billion annually on electricity. So we're talking about half the spend of all residential electricity across this country. And so I think it is safe to say that we have entered the utility era of computing. Now it just so happens that the utilities are Amazon, Microsoft and Google. Meanwhile, meta, first of all, doesn't want to buy power from those three utilities. They also use so much themselves that they want to build their own and have no interest in selling access to it. They just need their own data centers to run everything that they're doing in house at such an immense scale. And so we end the 2010s with a very mature cloud, right? I think it's almost in the background that people don't even need to talk about and explain the cloud anymore. It's like, of course, this is how you're going to start and run your company. The functionality is all grown up. All of the companies are now running containerized Kubernetes things. Your services are portable. Yes, they have their functional differences in their sales differences, but they're all kind of mature and stable multi region, all of this stuff going on. Nobody asks, do you have a cloud strategy anymore? It just is the way business is done. You also have the new startups that have built out the missing pieces. You have the snowflakes of the world. They're helping building more specialized databases. You have data dog helping you with orchestration. All significant public scale businesses that have built the missing components of the cloud moment. And the feeling right now is one of maturity of the cloud infrastructure, a maturity of a new large scale data center build out. And at the same time, a huge pressure to continue to build. There is a global race. It is fierce. It is competitive. And it has many, many, many more players than just these hyper scalars to give you a sense. Microsoft is scaled from 35 regions to 75 regions over a period of a couple of years. And so they're adding an entire region a month. This is data center region. buildouts. Deal sizes ballooning at the same time, a decade ago, a large lease might have been five megawatts. And you fast forward to the end of the 2010s, and every one of the hyperscalers are reserving 100 megawatt campuses. So a 20x growth in the size of the data center over this decade. These two big points fit. The whole idea that the cloud now can be taken for granted and visible is because of these scaled global buildouts. You don't get to do that if the thing isn't just working as usage is exploding. The entire industry is matured around a private equity money has come in. You've got developers treating this as a real estate asset class and their stockpile land, their pre-building substations ahead of demand because they know what the hyperscaler or the next large company is going to need and they're going to snap up that capacity. And so this buildout is fierce on land. Not only is it fierce on land, it's fierce in the seas. It's not enough just to have your own servers and your own buildings and your own power infrastructure. You need confidence that you can stay connected. The best way to have confidence is to put your own cables underwater. And so instead of just renting capacity, which of course they continue to do, but Microsoft, Meta, and Google and others start financing and building their own undersea network. And this way they can control end-to-end. You think about Google's vertical integration. It starts with the box, it goes to the rack, then it goes to the building, then it goes to the campus, then it goes to the wires that connect the campus. That's how you get confidence in what you're building and improved economics and improved performance. And performance continues to drive forward. But by the end of the 2010s, PUE had plateaued around an incredible 1.1. Again, only 10% of the power being used in this facility is not directly for the server and IT equipment. And so as we talked about in the sustainability race, the bragging rights had shifted. The marketing teams had moved off of PUE and now they're talking about carbon intensity or CUE. Now we're talking about water intensity, WUE. And the holy grail metric here is who is launching 24/7 real-time renewable-powered data centers. And everyone's putting out their climate pledges, building into their plans, and driving the entire renewable energy and PPA world forward through this genuine focus on the carbon intensity. The cloud by 2020 was doing what needed to do. And so we now had this scaled, matured global technology for storing things, computing things, connecting all of us, streaming content, powering video calls. Why might that be helpful as we go into 2020? Well, Ben, we're now upon everyone's favorite moment in recent history. The dark days of the COVID pandemic, a time where we all had to stay at home, and a time where society, systems, public health, everything was stretched to the max. The same actually goes for our data center infrastructure. While the 2010s saw fast, consistent growth of cloud computing, COVID compressed five years of adoption into about 18 months. With everyone at home, video calls, gaming, online collaboration, telehealth, e-commerce, SaaS, everything surged all at once. And we all remember the endless video calls, but let's put that into perspective a little bit. Going into this era, Zoom had something like 10 million folks as daily meeting participants. Fast forward just to March 2020, they hit 200 million in four months and four months. One month later, 300 million. So there's explosive growth. Google meets similarly, three million new users per day in April. PQC's was up 30X since January. And gaming was exploding. They're at home. They're looking for entertainment. So steam is shattering records. Netflix and YouTube famously had to throttle down the network speed to lower resolutions to avoid essentially breaking the internet. I was at stripe at the time. And to the e-commerce point, it was just this explosive moment where sure you saw some businesses perhaps and travel, in particular, struggle and go under. But the majority of stripes users were exploding. Everyone was ordering stuff online, Instacart and DoorDash. It's easy to say five years of growth in 18 months. What does it actually feel like when you're working on the infrastructure? It feels like everything is breaking and you're trying to make sure no one in the outside world feels it. And we managed to accomplish that and very proud of that. The API stayed up and payments were processed even as they scaled rapidly. Another company in the crosshairs and seized the day was obviously Zoom. They had to scale infrastructure from 10 million to 300 million users in months. They added servers in co-location metros. They used AWS. They used Azure. They expanded aggressively on Oracle Cloud infrastructure. They were just doing everything they could to ensure that those video calls didn't have delays. Under the hood, the way Zoom works, you can think of that split into two. Zoom control pain, which was making sure that it would respond to your logins and understand what you are as a participant in a meeting. But then you actually had the video feed. That would have these meeting zones where Zoom would make sure to connect you to the closest meeting zone in a carrier dense building. So it's kind of very similar to the CDN networks that we talked about earlier with Netflix, but had this dual bidirectional nature to it. They were in a race to add more of these purine machines in those telco hotels that we talked about earlier. And Zoom became proof that Cloud's elastic promise was real in the greatest stress test of its times. The capacity to materialize as fast as humanity demanded it. This was the moment we're in exploding demand. And there's a few key themes that affect the data center world in this moment. One of them is famously known as ZERP. So on March 15, 2020, the US Federal Reserve cut its target rate to the range of zero to 25 basis points. What does that mean? In the infrastructure world, that basically lowers the hurdle rate for your risk, for speculative bills. So now, campuses and shells can be built out way ahead of leases being signed. Because the carrying cost of that infrastructure was so low because you're caught your capital so low. You're buying a house. This was the time when you're interested on the mortgage, was as low as possible. So you could buy more a house. Well, if you are a company trying to build data centers, it's the same story. You can buy more data centers. Your same amount of principle is going to go much further in building out more. And as a company in this infrastructure environment, you can take more risk. The hyper scalars took advantage of this and started pouring money into building data centers. But so did Blackstone. So in 2021, Blackstone purchases QTS Realty Trust, one of the largest data center builders in the industry for $10 billion. There's so much capital, photograph and the beauty of the data center asset class was that it has started to look very predictable. It was looking like utility style returns. Everyone's investing in the reeds and the infrastructure debt. This is fantastic for the counties that were building this out, namely Loudon County, which we've talked about before, right? Their budgets are suffering in the COVID era period. But because their data center alley, they saw their server equipment tax go past $400 million annually. And so that prevented them from needing to raise property taxes to fill their budget gaps. I think at some point their vacancy was below 1% in all of this. So there's just so much demand for space, co-ocation, interconnection, everything is just flying off the shelves. Another major theme here is what is defining the build out. And what you find here now is that it starts to grow that the panic around the urgency for electrical supply. And it was both a amount of supply of power that you want, as well as a time to power. Because there was this capital looking to go to work, the question was actually how soon can you get me the power? This is new, right? The scale of the data centers earlier, this five, 10, 15 MW data center, didn't necessitate a big conversation with the utility. We're now talking into the 50 to 100 MW bills. And this changes the dynamic with suppliers. We had folks in Ireland saying for the first time, we're going to pass regulation that says the utility can block a data center from coming in, because it's too much load. You can't just walk up to a utility and plug in for 100 MW. For the first time, grid capacity and planning was a throttling factor in how fast data infrastructure could grow. The era from 2010 to 2018, which was this big shift to the cloud that we talked about, was one of increased efficiency in these systems that ended up bouncing out the increase in demand and capacity. So there was actually relatively flat power use growth despite there being so much growth. While there was more demand, it was the steady increase and the efficiency jumps could go in concert with the growth jumps. So you buy a new phone, you buy a new computer, most of it's all the same, except you just get more processing power because of Moore's Law. And that brought about these efficiency gains. And so we were building a lot of new data centers in compute, but total energy consumption of data centers actually remained relatively flat throughout this period. It was largely due to a few factors. One is that we moved into the cloud. And everything we talked about earlier with utilization then can come into play, right? Exactly. One study found that the cloud is 93% more efficient than running your compute power on premises because you have less waste assets combined with everything we talked about with PUE and driving down that efficiency. Well, now all of a sudden you had COVID plus ZERP where you're not waiting to get more efficient. You're just trying to build more and more. We'd also kind of run out of that low hanging fruit, right? PUE is at 1.1. So if you look at the big four that we talked about before, Amazon, Microsoft, Google, and Meta between 2017 and 2021 and those four years they doubled their energy use to 72 terriot hours in 2021. So much greater demand combined with the loss of the PUE benefits in the cloud efficiency means we're in a new paradigm of power usage and the industry is having to reckon with that. It's pretty amazing that we had such a flat period for how much growth that we just talked through. I mean, the whole scale of the cloud led to wow growing usage, the efficiency gains were so significant. We've run out of those tricks, right? Right. We, the hot and cold aisles are contained. We've virtualized everything. We've containerized everything. We need to come up with some new tricks. I hold out some optimism that necessities the mother of invention and we'll figure out some new tricks because we'll have to. We have to. That being said, there is a new power user showing up in an extreme way in this era. Let's talk about the crypto miners for a bit. Let's talk a little bit about crypto because it's not just a hobby, but actually ends up being a competitive buyer in the data center infrastructure build out by 2022, 100 to 150 terriot hours. So that's almost 50% that the rest of the data centers is now going to crypto mining. Go back to 2009, Bitcoin launches. It's pretty much just a dark corner of the internet thing, right? Someone mining on a PC in their bedroom. Then finally, folks figure if you buy the GPUs off the shelf, you can go faster. And people started even designing custom A6 that are designed to do nothing but mine, Bitcoin and other coins. By the mid 2010, mines had become significant power loads and folks started to build these out where they could find cheap electricity. And while the technology behind crypto was interesting from a business perspective, it was less of a IT business and more of a power arbitrage business because it was all based on how efficiently you could mine and where. You're doing a hard math computation as many times as you can, which is purely solving problems using power. And this is very, very different than everything we've said about data centers before. This is just compute as cheaply as possible. And this really comes to a collision point in the real world where you had markets like Iceland and Eastern Washington where Bitcoin miners were outbiting these cloud providers for the cheap hydro and the power that was available. Then the crypto industry starts to realize that there's power that's not useful to others. Let's look for ways and places to put crypto mining that you couldn't even put a data center or a normal industrial load. You're in an oil field and you're flaring excess gas. But what if you put a turbine there, you burn that and you power crypto mine. This is how Crusoe, which later becomes one of the premier AI data center buildouts gets started finding that cheap power source. And so it's interesting how many of these power related invasions came from the crypto moment. Then crypto loads were different. They were large, but they were fairly mobile. So that was an advantage in terms of where how quickly they could place. But it was also meant that policy could step in and shape the industry fairly quickly. By 2021, most crypto mining was happening in China. There was the cheapest power. It was close to where the hardware is being manufactured. But in 2021, China cracked down. It's no more crypto mining. And so all the miners fled to Texas to Kazakhstan, to Canada. And so by early 2022, the US share of crypto miners back up almost near 40%. This is up from 3 to 4% in 2020. What started out as a hobby and took over the fintech world and financial infrastructure, just real estate power and the compute industry to wrestle with how to connect very large, very flexible loads to stressed grids. And what innovations can be born out of trying to wrestle with that. And on that point, crypto as an industry didn't care much about its footprint. But the rest of the industry and the data center in particular continued to be very focused on the carbon intensity of their operations. And now these hyperscalers were tracking their carbon footprints with extreme precision. By 2020, Amazon became the world's largest buyer of renewable energy. It's a scale thing. Amazon signs the climate pledge. And for Amazon to hit net zero, they don't just have to decarbonize their data centers in their offices. They have fulfillment centers. They have travel of unprecedented scale, delivering all of the packages. And this is beyond just the quote and quote right thing to do as we've talked about. Sometimes the cheapest, most economical way to get new load onto the grid and scale is through renewable power. And in a zero interest rate environment, it actually makes these PPAs more and more attractive. And it lowers the hurdle rate. So a wind project that might otherwise have been too expensive in this environment is profitable and is able to be signed on to a Microsoft new build out. It's hard to appreciate the flywheel of more demand for the data center services driving data center build out, which can be funded by a zero interest rate environment, which then all drives all of the renewable energy build out, which is similarly funded by the zero interest rate environment. You just had an infrastructure build flywheel running that set us all up for what might come next. It was an extremely timely moment in the evolution of data centers and it actually served as a dress rehearsal for the unprecedented scale that we are going to see in the AI boom. This moment during COVID with cheap capital flooding the system, driving a massive new build out power, starting to become a much more limiting reagent in the development of our infrastructure and a breaking down of our global supply change, which was feeding this build out all happens at this moment right before a massive boom. Now we've spoken about the companies building the big infrastructure, the big servers, powering our life. We have a $4 trillion elephant in the room that I think it's time to introduce. Now time to talk about Nvidia. Nvidia started as a company that was trying to figure out how to make gaming run faster on computers. They made gaming chips and they did this for years and years and years. I remember them recruiting out of the Stanford double E department, many a student to go to work on gaming chips and they're very much tied to the gaming industry. Their growth was correlated to it. But it turns out the same thing that you're doing in a game, which is doing a bunch of math to figure out how to render a pixel on a screen is what you need sometimes to do complex science. A bunch of tough math problems at once. And if you're going to build a cryptocurrency, what are you doing? As we said earlier, a bunch of tough math problems at once. A lot of compute. Use multiplication at scale to such an extent that their stock price starts to get tied to what's going on in crypto mining. First they were been tied to the gaming industry. Now they get even more correlated to crypto and come 2018, you have the crypto winter. That fall in the crypto industry dragged Nvidia's stock down 17% in one day over the period they get cut in half. And so enter COVID and you hit a crypto boom bus cycle. But machine learning algorithms are spinning out more and more recommendations to us as we're sitting on YouTube and Netflix and Instagram. And you know what powers a recommendation model and Nvidia GPU. And so growth of these training models takes off. You can see it as a line item, right? And there are names. They have this carve out for what they call their data center business, which is really the growth of this business. You see 60% growth. So in 2020, Microsoft announces a next stage of their partnership with OpenAI. This time, OpenAI was not a household name. It was a research project, right? Yeah, I mean, that's a leading well-funded AI lab with this deep collaboration of Microsoft, building models that were available via API. And in the same year, they published a paper that really demonstrated something they had discovered and what they call their scaling laws showing that the bigger the compute was that you threw at the problem, the bigger the data set, you thought the problem, the better the model. And Google was discovering very much the same thing. The more chips, the better the model. How convenient. Music to Jensen's years, I think. Indeed. And so in 2022, right as the crypto crash is coming, you have OpenAI released a new model, GPT 3.5. It's a huge upgrade from GPT 3 and they had built it in their new Microsoft data center. And they're thinking about new ways to show to the world the value of what they've been able to build. Are you saying that everyone wants to like use an API sandbox to understand how good a model is? Right now, it's a little bit hard to access. But maybe if you put a simple box as a UI and create a little bit of a chat feature on the GPT and be nice to have. at least, right? Right. You know, few people might be interested. So they decide to put out this chat function on November 30th, 2022. The launch of chat GPT in five days, they had a million users. This was far and away over the team's expectations as well as their capacity planning. According to Sam Altman, they expected an order of magnitude less interest. By January 2023, chat GPT has become the fastest growing consumer software application in history. So after getting a million users in five days, they are now across a hundred million users in just two months. And by this last summer, chat GPT's website is among the top five most visited sites in the world right after Instagram. So there is real consumer and enterprise pull. And world's been waiting for AI to be a thing for a very long time. But this is the moment where everyone says, okay, this is it. This is the time. You end up with over a billion prompts a day. People are loving this thing. And everyone knows, well, to make it better, we just need more chips. Why is Jensen so delighted? Well, where are we now? Three years later. More chips, better models, right? Three years ago, in 2022, Nvidia's data center revenue was growing rapidly. Four billion in revenue that quarter. Last quarter, their data center revenue was a staggering $39 billion. That's in three years, it's growing nearly 10X of what it was. To add 36 billion of quarterly revenue in three years, it's unprecedented. This is powered by this discovery of the scale loss. And the timing of their product launch could not have been better. They had the A100 that was essentially the kind of like training, scaling your training, but they launched the A100 in 2022. That was designed for this moment. It was the rocket boost of a chip. It added new special math modes tailored for transformers. It added more high speed memory. As a result, it's slash training time and made inference faster. And these A100s are flying off the shelf from selling a million and a half in 2023 to over 2 million in 2024. And the big players having learned from the past years of the cloud boom and the COVID demand boom, they're ordering these in the hundreds of thousands. And these things are not cheap. Each chip is like a car, right? $40,000 a piece. Supposedly, you couldn't buy one chip at a time. No, why would you sell one car at a time when you can put them in boxes? They really sold in two models, right? There's training boxes and inference boxes. And the training box is an eight GPU box with a CPU in the middle. And they're hardwired with this NV link connection. What it's really interesting about this is it's all about making it effectively one big GPU. It's all about creating this really high bandwidth connection between these GPUs because when you're training, that's what you want. An NV link, if anyone's ever built a computer in their day, you would have what's called a PCI connection in a motherboard. The way that you're Nvidia graphics card would plug in to your computer was using PCI, felt like there's a really fast connection. Well, it's not fast enough for this. So Nvidia invented NV link. It's up to 15 times faster than that connection. So these GPUs can communicate extremely quickly. One training box, it's about 500K to buy one of these things. Then they also launched these inference boxes. And so this is two GPUs coupled with a bunch of memory to be clear. Infraints is when you're asking chat GPT a question and it's responding. That process of responding is inference. You want a lot of memory to be able to understand all of the content in that response. These boxes were designed with a lot of fast memory connected to be able to share context across the model. Now, selling these GPUs in a box actually has a physical space implication. So you've heard us talk about the racks and racks in the data center, as far as the I can see. Well, these racks have slots for servers. And those servers have a traditional power consumption. But when you put in a specialized server, it starts to change the power consumption of that rack and therefore of the square footage and of the data center. So let's unpack just for a moment what's happening at the rack level when AI arrives. Historically, one you rack would be made 400 watts. And so you'd build a rack with 20 to 30 of these servers. You'd have some networking equipment. So somewhere around five to 10 kilowatts of power going to a rack. That's like 10 hair dryers running at the same time to give you a sense. Now, the GPU boxes arrive. Those each GPU training boxes. Each of those is up to 10 kilowatts by itself. So it's the same power consumption and heat as the whole rack was. But you could now pack a rack physically with 48 of those boxes. So all the sudden, you're an order of magnitude up in power consumption up to 90 kilowatts in a rack. This has a bunch of implications, not just for power, but also for cooling. An inference box is somewhere in between. So a whole rack maybe is up to 40 kilowatts. One of the things driving this power conversation we've been having is again, not only the size of these data centers, but now the density of power required within these data centers. This is what makes an AI driven data center so much different from a traditional data center. Just 20 of these training racks in a pod is one megawatt of power. That's on the order of a small 800 person town powering electricity. So you get 200 of these racks. You get to an 8,000 person neighborhood. Very quickly, you get to a gigawatt of Seattle scale power. As you pack these GPUs that have higher power density, they're running hotter. And you can't let them run hotter. So now how you cool your GPUs and your racks needs to evolve. The problem is air can only move heat so quickly. And if you were trying to do that even with the inference racks, you would need leaf blower level fans to move the air out. And so you need to start bringing more cooling down to the rack level. So the first thing that you can look at is retrofitting the racks with what's called rear door heat exchangers. This puts some water cooling directly alongside the rack. So that when air is moving over those hot chips, some of that air instead of just going straight into the hot aisle can get extracted via some tubes of water running inside the rack. That is a nice convenient half step because you don't have to change how you're actually cooling the overall system. The next jump is the jump to direct chip liquid cooling. This is where you mount cold plates directly on your chips and you run water right near the chip to actually extract the heat away. Instead of expecting air and heatsinks to do it. And so increasingly, especially with the most powerful GPUs and to be clear, each generation of GPUs gets more powerful and therefore has more heat to remove, we are moving to a water cooled world. We've crossed the threshold of physics where air can cool this amount of heat being emitted at this level of density from the racks. And so now it is a liquid cooling world. Now at the end of the day, you've got to move the heat from inside to outside. But it's very environment dependent. In the right climates, you can kind of expel it outside. And that's why we saw some of these data centers looking at Finland and Oregon and certain geographies that enable that to happen. And this consumes very little water. But in other climates, you have a large evaporated water cooling towers. This is where the problem of the water impact on the local community starts to come into play. Fundamentally, what we're trying to do is we're removing heat through that phase change, right? You're evaporating water. That's a massive transfer of heat. These towers increase the surface area of the water. Water evaporates out. It looks like steam going up from these cooling towers. And it's a very effective and efficient way to cool things down. But it can use up your water. And so you end up with this energy water tradeoff in certain environments. And that becomes a big planning focus, not only for data centers that are working with their local community, but also those that are power constrained and built by companies that care about carbon intensity. One study Google did in 2022 found that water cool data centers use about 10% less energy, which means they emit about 10% less carbon emissions than many of the air cool data centers. Now the problem is you can't do this as easily in water stressed areas. So there's essentially this knob you can dial up and down to be clear. It's possible to build data centers that don't use any water, but they're going to use more electricity to run your, you know, an air conditioner in your house. You're not using a bunch of water, but you're taking a ton of energy. And so I would argue if we care about water, the best thing we can do is have clean electricity to make this an easier trade off to make. So if you zoom out and look at water globally, data centers consume over 550 billion liters of water annually, which is a big number to put that into context. A single 100 megawatt data center can consume 2 million liters of day or the amount equivalent to 6,500 US households per day, both direct and indirect water usage. So this is a very big real and current issue for data centers in terms of where they cite their data center, how they get permitted, and the ongoing operations. The local question is hyper important. Some reporting that the New York Times did about the situation in Georgia, where supposedly when Meta broke ground on their billion dollar data center built out, water taps and some residents nearby went dry. And you know, there's some back and forth around whether or not you can prove direct causality. Still working out whether or not that is actually the case caused by that data center, but it is known to be true that that that does and our is using about 10% of the county's total water and it's driving water prices in the region to go out. So now let's dive into what is the largest constraint to the data center buildouts today. And that is power. We've talked about kind of the evolution of power moving up the decision stack. From the early 2010s through the COVID era and now into AI era. Power is really the limiting reagent. The image in my mind is we have the largest most complex machine that humankind has built in the power grid. That has brought us such evolution and how we live our life. And it is colliding with the new complex most interesting machine of the data center. In this collision and the force of it that we're living through right now is defining this period. And it is no small collision. It's a function of two things right it's a function of speed and a function of scale. So as we've seen, there is a absolute arms race on compute because the faster you can get compute online, the better your models are, the more revenue you're going to make. And timing matters a lot here because a model trained in 2025 can become obsolete by 2027. So if you have a two year delay in getting your data center online because of the power grid, it becomes a deal breaker. It has to happen now coupled with the scale of these data centers. And so we've as we talked about a typical data center started as five to 10 megawatts. Then you know, just a few years ago, we were in the 50 to 100 megawatts. And now this year we're starting to see gigawatt scale announcements. Now to be clear, these are campuses that make up a gigawatt, their multiple facilities, but they are still being built out cohesively and will put a strain on the grid and the local community at a gigawatt scale. Now to put a gigawatt in perspective, this is the scale of of a cities power consumption, right? Maybe a Pittsburgh or Cleveland and Google's data center electricity used doubled over the four years and was up to 30 million megawatt hours in 2024. From 14 and a half million megawatt hours in 2020 to put that all into perspective. That's about three million us homes or around three quarters of a percent of all US electricity consumption, which checks out right if data centers are currently about four percent of US electricity use. Because it makes sense that Google's about a fifth of that. And it's worth pausing there for a second to talk about four percent electricity use. When I hear that number, it actually seems shockingly small for all the discussion of data centers and electricity. That's okay. It's four percent. Let's say it goes to 8 percent like compared to industry compared to, you know, cooling and heating buildings. It all feels small. So why might this be such an issue of conflict? It's a great question. It's it might have to do with the fact that it's misleading to think about it in terms of total national electricity use. Because a data center has a localized and concentrated impact. Right. You're not spreading this load across multiple utilities. And not only is it localized and concentrated, it's localized and concentrated in similar areas. Because as we've been talking about the network effect of the value of a data center being positioned near the undersea cables and near other network points is where you get a lot of performance gains. And so you end up concentrating yourself in Virginia in California in Texas. There's just basically 10 states where you're seeing new data centers come online. And so there's a tremendous impact at a local level at the electricity prices for that community, that county and in that state. But not necessarily at a national level. You know, we're talking gigawatt scale. That is all about these large training clusters. When we're talking about inference, which is, you know, hey, chat to PT processing your response, you actually probably want that spread out, right? You want that closer to the edge. And that's where you need it, you know, close the interconnect because you want it to not have latency when interacting with the user and their data. So let's talk a little bit about why you need a gigawatt data center or five gigawatt data center to do the best training possible. You don't really want to think about it as one GPU or a rack of GPUs or a building of GPUs. You want as much as possible the whole data center to act together. Because the way that a transformer actually trains is it, it makes guesses about what should come next and then it compares that to reality and then it tunes the weights across it. And you're doing that collectively across the whole model. And to do that, you need all of the computers working on the problem to be able to communicate quickly. Otherwise, the whole thing is training slowly. The faster those exchanges can happen, the better. And so you want the connectivity between chips to be as fast as possible. You want the connectivity between boxes of chips to be as fast as possible. And so it does not work if all the sudden half of your computers are on these coast and half are on the west coast because the speed of light across the country is going to slow you down by multiple orders of magnitude. Then, you know, within one campus within one center. And that's why meta wants to build a five gigawatt campus because that's going to get them the biggest training model possible. There's also an interesting nuance here where training models hit higher load factors than an inference model and a typical data center. You're trying to run this all at once and you're utilizing ideally all your expensive chips. And so all of that leads to a higher utilization and more power and more compute. So we talked about the speed more time to power is time to revenue. We've talked about the scale now going from 100 megawatts to gigawatts. So this creates an objective function when we're thinking about the power constraint on data center growth. And that ends up being how do we secure large 24/7 power where you need it when you need it quickly and ideally as clean as possible. Should be easy, right? Oh, we're not to mention there's supply chain bottlenecks and transformers bottlenecks and gas turbines bottlenecks and labor and specialty skills. We don't have the most up to date, freshest grid with transmission lines. So this gets into a very tense moment and an area that breeds arguably innovation. So the reality is getting affordable power that is 24 by seven quickly and cleanly is going to be very rare. That goal deluxe situation doesn't really exist today. And so we're going to have to compromise on at least one of those variables. Now, if you got a few different options to get these data centers online, right? You can either hook it up to the grid and use power that already exists. You can build new generation of power that's off the grid and power that data center, do some sort of hybrid in between. And over the long term, we can build out large new generation like nuclear, geothermal and others. Watching this up closely, there's both a lot of moments of concern, right? Because we're kind of at times doing things like keeping a coal plant running longer than we might have otherwise. But there's also a lot of interesting moments of innovation, a lot of growth in storage to help supplement what's going on and smooth out power generation and co-locating these in new hybrid ways. And there's a tension that's that's largely dictated by timing, right? There's a near term pain that local communities are facing when you bring a data center online and potentially increase electricity prices. But there's also caused for optimism because you've got well-funded companies who have an interest in long term sustainable power. And the fact of the matter is the most affordable long term sustainable power we have is going to be clean and renewable. And so this moment actually offers an opportunity for us to accelerate a lot of new clean build on the grid and reshape our electricity infrastructure. I think at the end of the day, there's also a big issue of incentives. These high-first scalars, they want to build these models now. Many of them are used to mostly sold building their businesses on software-based timescales. And the reality is, despite all the discussion of the utilities and electricity, for them, it's a bottleneck. The actual cost of energy for them to build the biggest models is like two to 6%. It is all in the people and the hardware. It is all in the chips and the humans. And so for them, it's all as you said earlier, and I, this speed to getting out there. And that's not how our system is set up. That's not how our incentive structures are set up, right? That's right. Because the utility supplying that power operates on an entirely different paradigm. And that typical electric utility, power delivered in 2027 is valued at the same amount as power delivered in 2030. Utilities don't differentiate from those products. But the customer, a Google or Meta, massive difference for their business. They need power today. How much more would they pay if they could get power now versus three years now? And how much could we unlock in terms of investment into our utilities if they could take advantage of the fact that companies are willing to pay a lot more today than in two years? Brian Janus calls this the what bit spread. It's kind of the economic arbitrage between an available what and the ability to turn that into a bit reality is the speed of regulatory change in utility, tariff pricing, which is a fancy way of saying how quickly the utility can change anything about their economics is slow. And so despite there being like an obvious market demand structure, it's not going to change in the next 12. months the way that utilities price their product. Utilities are public goods, right? They power our lives and our schools and our hospitals. And so for good reason, they are regulated. But because they are regulated, they take time to change. And moreover, we have a federated system. We have some 3,000 different utilities across the United States. And so making that change across this country, or even in the 10 states where data centers are getting built out, will take a lot of time. And power has become the dominant question of where can you build your data center because you can't go do it without that. But it didn't remove the fact that the other questions still remain. For example, you need people to go build the data center. You need a lot of electricians. You need all the components and the supply chain and the thing to build the transformers to step down the power. Based on how we've been talking about this, you might think that data centers are almost entirely a US story. And while the US is the dominant builder of data centers, there is actually a global story here and global infrastructure being built out and a strong need for sovereignty for other nations. So in order to help understand what's going on here, let's go back in time a little bit to understand the drama that's setting in globally. I think it's worth looking at just fairly recently. In 2018, you had Europe past GDPR and that set this world baseline for privacy. Thank you all the cookie pop ups. But also really tight controls on where EU data sits and what happens if you have the EU resident data outside of the EU. And so this is really, I think, a reaction to where are people storing data about our citizens. And so there's controls about what happens if you're holding a transferring data outside of the EU. In 2018, the US passed the Cloud Act, which essentially lets US authorities lawfully demand data from US providers, even if that data sits on servers outside of the US. And so the net effect is if you're working with EU data at scale, you start need to start having servers and hard drives in the EU. And this becomes, I think, almost another lock in that the big cloud providers start to have because they go and build the infrastructure to do that. And we saw this in the Microsoft story, right? A big part of their early scaling was following their enterprise customers globally and needing to build out data centers in Germany as one of the primary cases. And so the EU, you know, US looks like a little TIFF drama. I think it's time to move to the major geopolitical data center drama, which some have deemed the new Cold War and there's a book called the Chips War, which centers around this notion of the US and now China in a fight over everything. But centered on this technology, right? So I mean, let's go back to 2012, which I think really kicked off this last decade, 15 years of animosity here. The House Intelligence Committee report warned US carriers away from using Huawei and ZTE equipment saying that we shouldn't use this in any sensitive government system. Now that of course starts to spiral outside of this government use. That was first with the carriers Beijing responded on a different front, the data front. So in 2017, China's cybersecurity law, followed by other laws, pushed data localization and tighter state control. And so the US companies had to respond by localizing operations, which essentially means AWS, Apple, Azure. They had to sell off to Chinese partners or find a Chinese government linked company to be a partner. That's right. So it's not just that those companies, hard disks and servers had to be in China, but they actually had to be owned and operated by a Chinese company. And so then the US, you know, react even further, they borrowed all federal agencies from Biden or using equipment from a long list of covered Chinese vendors and they borrowed the FCC from approving any new authorizations from Huawei's ET and others. So this just ramped up further and further. This brewing war was not just limited to land, but also involves our friends, the undersea cables, right? And so in 2016, Google and Facebook partnered with a Hong Kong based company to build the Pacific light cable network, a massive 12,000 kilometer undersea cable linking Los Angeles to Hong Kong, Taiwan and the Philippines. Part of that global network of undersea cables, we were talking about earlier. It seems like a great idea. What could go wrong? And fast forward four years later, 2020, the US national security officials say we don't know about this Hong Kong landing point anymore. We think that's going to be a vehicle for Beijing to have surveillance across this important back when the internet. And so the FCC ultimately blocked that route. And so now the cable hits Taiwan and hits Philippines and those are lit up, but the Hong Kong branch, it's been laid, I believe, is just laying dark. So I'm sitting here in Los Angeles without that cable to Hong Kong. Now moving back to land and to chips, the battle continues. And so in 2022, the Commerce Department rolls out these sweeping controls on advanced AI chips and fab tools. And so remember, this is the moment where we've gone through a COVID boom. We're seeing an uptake in usage from crypto and streaming and now AI. And the Commerce Department tightens these controls again in 2023 and again in 2024. And this feels very active, right? I mean, literally every quarter of Nvidia's earnings, there's discussions of this. Few weeks ago, the US opened a channel that said in video name, D, Censorship, certain chips that are dialed back like the 20, but they have to give the US government a 15% cut of the revenue. It's a pretty unprecedented pay to export arrangement, like a lot of things that are unprecedented that are going on. If you zoom out, it raises some good geopolitical strategic questions on how we want to engage, right? Yeah, it's foundational to the new economy that's being built on this advanced compute power. And it remains to be seen how this plays out. It may backfire, right? China is not terribly keen on being dependent on US production. So they are infusing mass amounts of capital in their chip sector. It's still a step behind Nvidia in terms of raw performance and software ecosystem dominance. But given China's resources, you have to assume that that gap is going to narrow and narrow quickly. There's a reason China wanted domestic energy. They want to have their own domestic production of silicon here. And so they've invested heavily. And they're catching up, right? I mean, some estimates that they're 60 to 70% of Nvidia performance. It depends on what they're doing. But they're investing out of a $50 billion fund to improve chip development. And even more so, they're passing policy that says data centers need to source at least 50% of their chips domestically. So they're making sure to really stand up not just the supply side of the market, but also simultaneously the demands out of the market. The story is less about whether Chinese chips can equal Nvidia's top end designs and more about how quickly that's going to happen, given the demand and the capital they're putting in. In short, at the moment, the US is still ahead, but the China chip ecosystem is accelerating rapidly. And their backlog of demand for Nvidia chips, which is being withheld, is going to matter less and less and less over time. I mean, the other dynamic that you have geopolitically is governments are seeing the AI data center boom as an important future part of their economy. A deal that exemplifies this is the Emirates deal with open AI, you know, around this UAE Stargate build out, right? A one gigawatt AI cluster in Abu Dhabi and all the ecosystem players more called Nvidia to Cisco to soft bank coming into finance this and collaborating on the build out. And I think it just raises a lot of interesting questions, you know, what about the location of a data center matters to a government? And the map is stark, right? If you look at where AI data centers are located today, only 32 nations have them and most of them are in the northern hemisphere. You have large swaths of Latin America and Africa as fully dark and governments are deeply, deeply concerned because they are wondering that if you continue to rent compute power from far away data centers, you remain at the whims and you remain vulnerable to foreign entities and foreign companies and you aren't able to support domestic enterprise or domestic scientific researcher, academia, with the same level of control. And so this idea of compute sovereignty is top of minds for a lot of people and a lot of emerging markets are worried that the AI era runs the risk of leaving them even further behind economically. There's almost a rough analogy to energy independence, right? And like, do you have it regardless of your relationship with another country? And that feels like the important threat here. The other tricky dimension here, which you've talked a little bit about, is the actual impact on the climate, which gets a lot of attention these days, right? So, there's a few questions that continue to come up and one that Ben maybe your mom has asked you and her mind has is if a chat GPT query is bad for the climate. If I care about the climate, should I really be using AI? And we have some data on that. There's been some reporting on this and I think it's an area that's become a pretty hot button issue for folks and there's been, I think some people even shaming others for using these tools because the climate impacts, it's worth looking at the latest numbers. Google actually just put out a publicly readable paper on gemmins. where they go down to the details on the energy consumption and the carbon intensity of that energy. And so the median Gemini text prompt, which for all intents and purposes I think could be viewed as fairly equivalent to a cloud or a jet TBT or a co-pilot prompt, uses 0.24 watt hours of energy and emits 0.03 grams of carbon dioxide equivalent. And consumes 0.26 milliliters or about five drops of water. Those are figures that are substantially lower than what the public estimates have been. For some kind of sense of scale that per prompt energy impact is about the same as watching your TV for less than nine seconds. Why do we think the public reporting might be so far off? There's been a lot of improvements over time. And so it's important to kind of ground ourselves in this moment. We know through that same report that the AI systems that we're using are becoming more efficient. There's constant innovation in the software and the hardware to drive more efficiency. So over 12 months, the energy of a standard Gemini text prompt dropped by 33 fold. 33 x improvement in the energy consumption even more on it from a total carbon footprint per century, 44 x all while delivering higher quality responses. And so we are moving in a world where the efficiency gains are continuing. And this question of is your query bad for the climate is fairly insignificant. Hannah Richey from our world and data has independently corroborated this fact. It's an interesting cocktail conversation, but largely misses the forest from the trees. I think the pressure is a good positive pressure because one individual query should not feel bad about. But the pressure for all of these companies to monitor this and drive these numbers down and get these efficiency improvements is a fantastic thing. That's right. Right. And drive the deployment of clean energy so that the carbon intensity of the energy drives down is a good flywheel to keep pressure on indeed. So mom, keep asking question because Google will then continue to improve. I think there's another area that's been under discussed. You don't hear a lot of discussion on the actual carbon intensity of building these very large pieces of infrastructure, pieces of real estate. Let's call it embodied carbon. And so what is the embodied carbon impact of the steel the cement the energy used to build the shell and stand up the facility in and of itself. Thankfully, Google did another report on this and shared that in their analysis, running an AI data center, the operational emissions, which really means the energy going into running the data center ongoing is going to be about 70 to 90% of the total emissions. Manufacturing emissions, which is really the manufacturer of the server components, the memory, the flash storage, the GPUs is going to be around 25% and then the data center construction. So this is the steel and cement and the logistics around that are around 5%. So this is all to say, I think when you look at this, we should probably be in a bit more attention to the manufacturing of the components and make sure that we're looking at the LCA's for solid state drives and GPUs and making sure we're taking that to account. But the tensions probably correctly centered on the electricity. And on this point of focusing in on the manufacturing, you know, the procurement muscle of one of these companies that are building is out is actually a very big lever. So in Microsoft requires its suppliers to use 100% carbon free energy by 2030. It will pull their suppliers in all the the fab building and the motherboard building and the assemblers. It's moving everyone down the supply chain. It's called scope three emissions into a cleaner world, which which has tremendous power. I think the other thing that's interesting is some of the circular conversations are starting to happen. So I was out of talk where the Microsoft CSO was talking about how they recently launched. I think it was just in April this year, a circularity program where they would take the rear earth material in the hard disk drives that they're using in the data centers. And they have a new way to recycle those while disposing of the data and that yields a 90% recovery of those rare earth materials, which also helps stand up a US supply chain around rare earth materials, which we don't really have today, which is pretty phenomenal. Yeah, tremendous and it's going to be a growing discussion over the coming years as we mine for more and more of these critical minerals and the geopolitical control over who has those minerals. All right, so given all of that operational load is the main contributor to greenhouse gases, something like 70% or more of the emissions. How should we think about the scale of that from a meta climate perspective? So we talk about operations remember we're talking about the energy consumption and not the materials that go into building the data center. Recent studies show the data centers account for about 4% of total US electricity consumption and with more than half of that electricity derived from fossil fuels, that means the data centers generate more than 105 million tons of CO2 every year. So reading the study, I didn't realize that data centers carbon intensity is actually higher than average exceeded the US average by 48%. The big point here is how grid dependent that impact is right. If these data centers are located on a dirtier grid, let's say a coal heavy grid in the middle Atlantic region of Virginia, then it's going to have a very different climate profile than a data center that's in Eastern Washington that's entirely powered by hydro. So we talk about 105 million tons of CO2. How should we think about that? So a few comparisons to help think about the scale of 105 million tons of CO2 emissions. You know, one is you look at at a US aviation emissions. This is about half of that. Another would be looking at enteric methane, which is a fancy way of saying cal burps is around 178 tons. So that means data centers are about 60% of the equivalent of that or all of US passenger vehicles are 1000 million tons or gigaton. And so data centers are about 10% of that today. Now you look globally, this goes up by approximately a factor of three as do you know the rest of these things. So you know, overall data centers are not insignificant. They're worth tracking on the map here. But quite fractional compared to passenger vehicles, big emitters like livestock and a third to a half of something like global aviation. You know, the interesting thing I think what makes us a hot topic is not just the current scale, but the projections, right. And so if all of a sudden we double on our current grid in the US, if we double, then now we're caught up to the emissions of aviation in the US. And if you keep going from that and you keep the trend line up, you could imagine data centers becoming a fairly dominant story in emissions. Other leg of the stool in this conversation is how it's affecting the local communities that these data centers are being placed in. And so one of the themes that we've found come out through this is the impact of these data centers are concentrated and they are disproportionately felt by the environment, the ecology and the people that immediately surrounded. And so we metaphorically went to Memphis where XAI is building a very large data center on the banks of the Mississippi 15 minutes from downtown. And and their Elon is commissioning a 200,000 GPU unit data center. And there's been a significant backlash from the Memphis community for a good reason. I think a lot of it centers on how are we going to provide power to that because the utility said that they can provide XAI power for about 50 megawatts of load. But XAI wants triple that amount. That's a lot of GPUs and 50 megawatts isn't going to cut it. So they brought in gas turbines, 35 turbines, which could theoretically power for and in 20 megawatts. The problem with that, I mean, it's good they're solving their power problem. But the other problem with that is these are highly polluting gas turbines. And so they have the potential to emit a couple thousand tons of small forming nitrous oxides each year. For context, that's more than the small caused by the Memphis airport. So it's like you're adding another airport of smog to the region. And this impact is not continuous, right? Because these data centers have been flow in terms of how much power they are they are consuming. So it's important to look at these peak moments. So we found public satellite data from NASA and the European Space Agency that shows that on average nitrogen dioxide concentration increased by 3 percent when comparing for a year before in this area. But in the times of peak consumption, we're talking about a 79 percent increase in nitrogen dioxide concentration from pre XAI levels in that area. And so you imagine it's like sitting in a traffic jam and then those moments have outsized impact on on the community at that moment in time. And so there's been a lot of pushback from this pollution and health risk standpoint. It's hitting South Memphis neighborhoods that already have elevated as men cancer rates from past industrial waste. This temporary electricity generation infrastructure isn't going to sustain in the long term. So all of this, this is kind of a case study in both the local impacts and also the time to power issue. And this notion of time to power essentially what it means is there is enough demand to get these data centers online quickly. The companies are willing to pay more and more and more. It's many of that kind of economic pressure. The community resistance sometimes can only go so far. And so as we're seeing in this case, XAI despite the pressure is currently building out a second location a few miles away, which will be double the size of the first one. So to half a million of these GPUs, I mean just insane. That's the current moment we're living in. So far we've been. trying to keep it focused on what has been built more than what is proposed. But I think it's worth taking an extreme view on some of what's coming. Meta has announced their big long term project, their big training center, the Hyperon project located in Richland, Paris, Louisiana. It's a $10 billion buildout online by 2030. They'll have a few steps in between. The goal is for this to be a five gigawatt campus covering over 2,000 acres. That is 2,000 acres. So a different way to visualize this is this data center is about the size of lower Manhattan. It is at a scale that is kind of unfathomable. Right, that we're going to build buildings of that scale in a few years here to power such a new technology. That Louisiana project is a great example of what we are living. The step change function that we are living through right now. This is the biggest tech infrastructure project since the 1960s, the dawn of the computer age, or even the 1880s, the heyday of the railroad period. I think Nvidia is on pace to capture the highest share of market-wide capital spending since IBM peaked in their percentage of that in 1969. Right, and so a lot of comparisons are being made to other past booms like the Gilded Age or the Tullco buildout. And when you mention booms, obviously you end up thinking about busts. Right, and so we talked about the Tullco boom of the 1990s, which contributed to the.com crash and a bust on fiber. You also look back at the 1870s and the huge railroad boom that led to a crash. And both of these posed the question of like, did we overbuild? Are we outrunning our demand? In both of those cases, it's not that the CapEx spenders were wrong. They were just early. We saw this vividly in the fiber overbuild, right? By the year 2000, we were only using 3% of the fiber we had laid, but it was absolutely necessary to power the next couple decades. And that's what it looks like for AI is that there may be an overbuild in this moment, but the foundational nature of this technology suggests that it's going to get utilized. Yeah, I think there's such a rush here because the potential AI market creates this economic imperative to go and plug in as many GPUs as quickly as possible. And that will build the largest and smartest AI model, which we'll create a moat. I think there's an interesting question here, though, of what is the infrastructure? Because yes, we still use the railroad tracks and we still use the fiber. But remember that all the companies that were building the fiber and the ones that were on top of that, it was too early for them to capture that value. But what's in part different about this boom is how it's being funded. Now, yes, there's VC dollars and other dollars going into it, but at least historically, and if you look at the major capital expense, it is these big companies we've spent the episode talking about is Amazon, Microsoft, Google, buying from Nvidia. And they're using the fact that their business models have thrown off billions and billions and billions of dollars every quarter to then go buy the chips and build this out. So there's a real kind of concentration in some of the build out of the infrastructure. Those previous boom busts financed by the banks end up drawing in the rest of the economy. And while there is some private credit actors, venture capital actors, because it's the Rology Finance by free cash flow of these private companies, a bust hurts them, which will drag down the stock market, but is hopefully insulating the rest of the capital providers that are not involved. There is supposedly now more private credit starting to come in to find this and that could create linkages to other parts of the system. We will see how that plays out. We've talked about the story of data centers up until present day. Let's zoom out for a second and talk about just where we are today from a sense of scale. Let's start with the most basic question. How many data centers exist today? There are globally approximately 11,800 data centers worldwide. Now of course, the counting of this gets nuanced when you think about the closets that still exist with servers as they did back in the day. But by and large we're talking about independent construction. And the US is by far the largest with over 5,000 data centers followed by Germany, the UK, China and Canada. So the US has almost half the data centers? Wow. Yeah. Heavy concentration in the US. Obviously other countries are catching up China's building very quickly. So this begs the question of who owns all of these 11,000 data centers and who's spending these hundreds of billions of dollars. And you can kind of think of it in four broad categories that's representative of the US. But also it fits globally. And so the first category are our friends, the co-location centers, the ones we've talked about, equinex, visuality and others. There's about a dozen of these that are building 10 to 100 megawatt blocks in the US and around the world. The second category are the hyperscalers, the Facebook San Muzanz Microsoft. You can add an Apple Oracle in there. They account for almost half of global data center capacity and they're the lion's share of new growth. But importantly, you still have thousands of private server rooms in banks and retail facilities and public institutions that are out there. And so while large in count, they're relatively smaller in capacity, but today still it make up about 35%. And finally, you've got legacy telcos that are still owning and operating data centers. Now the interesting thing about this is the trend, right? You're seeing enterprise and public sector data centers trending down and new build increasingly going to hyperscalers and these purpose built co-location facilities. So how much actual physical real estate space are all of these data centers taking up? This is like anything that gets into this level of detail. It can be hard to pull exact numbers. But one reasonable estimate is that maybe there's one and a half billion square feet globally in data center build out. For context, if you took an average median US home of around 2200 square feet, that's around 730,000 of those homes of square footage or 28,000 football fields, importantly, including enzymes, that kind of feels like a big number. But for some comparison to some other infrastructure in our life, if you just took the US interstate system, so this is, you know, I five, I 90, the big interstates, just the asphalt of those interstates is 10 times the square footage. So we'll start with power. Remember the impact of data centers on power, it's quite localized and concentrated to its physical location. But let's talk about it in a sense of scale. And so we've mentioned that in the US, it's about four and a half percent of total electricity. That's roughly 17 million households annual use. How can I conceive of that power? Yeah, I think if you look at the electricity consumption of the city of New York, it's about three New Yorks to power all the data centers in the US. Or if you look at all data centers globally, it's about equivalent to the power consumption of the United Kingdom. If you look at other comparable industries, it's pretty close to the power consumption of the chemical industry or the primary metals industry, you know, this is extracting iron ore and smelting aluminum. Similarly on the water question, there's a tremendous amount of nuance in how to aggregate total water consumption. But let's just do the average of the average. Let's take direct water for cooling, indirect water from the power generated for the plant and get a sense of how much water this is using. US data centers together consume about 250 million gallons of water per day. That's about a quarter of what the city of New York uses. So we've talked about the space that data centers, how many of them who owns them, how it's being powered and cooled. But what's actually inside of them? Think about the scale of the compute power. And it's actually a little bit difficult to figure out what humanity's overall compute power is. But let's just say there's about 50 to 100 million servers globally. And so the best way I think to get at this was to go back to the electricity usage numbers and then look at how much compute power we get per unit of electricity. What's surprising here is that if you run the numbers, we've around 40 xd humanity's compute power over the last decade, but only have grown power consumption about two and a half x because we've gotten more efficient. And if you look at that new compute power leaks come online only around 10% of data centers are AI focused today. Of course, that's a growing percentage. So compute power is one part of what the data center is doing. Let's give a sense of the scale and storage. Yeah. So in the last decade, we've three to four x total installed storage capacity to around 15 zettabytes. So we're speaking to a friend of the show Byron who made me understand the modern marvel of hard drives. So imagine the head of the hard drive is scaled up to the size of Boeing 747. It's flying at 560 miles an hour. And it is about one piece of paper thickness above a football field. That hard drive head is reading and writing to every single blade of grass moving at 560 miles an hour. That is absolutely insane. And so we have all these hard drives we have all this compute. And now they're connected via these submarine cables. How many of those do we have now? We have approximately 600 active submarine cables that are powering 99% of international usage on the internet. So in total, we're about 100 x the global bandwidth that we were a decade ago. And we've seen internet bandwidth continue to grow 25% year over year. So these numbers are today's snapshot. Now there's a lot of projections out there of where growth is going to go over. the next five years, it can get fairly outlandish, but suffice to say growth is continuing to accelerate now and in the years to come. All right, well, we talked about Byron's plane, reading and writing blades of grass. Now it's time to land our plane and I, let's do the themes. What did you realize, bubble up for you over the course of this research and conversation? It's been a journey. I did not know a lot coming in and have learned way more about the ins and outs of data centers than I ever imagined. And a few big things will continue to stick with me. One of the first books we read is called Tubes and it's this nonfiction nerdy detective story to discover where this thing called the internet lives. It turns out the cloud is quite physical, but even more interesting, it's strangely concentrated. The internet has almost infinite edges, but it's got a shockingly small number of centers and has been built in this hub and spoke structure. And so for me that starts with May East, right? Ashburn, Virginia, home to 13% of global data capacity and has once carried north of 30% of the world's internet traffic. That is the undisputed capital of the internet. And it's so wild that became the hub just by kind of being the hub. There's just like this center of gravity that spirals on itself. You know, a black hole of data center investment. Almost by accident, right? Yeah. The internet structure is based on this mesh connectivity of global cities on coastal shores. It's Virginia, New York, DC, London, Paris, Amsterdam, Tokyo, Seoul, Lagos, all connected by these 600 undersea cables that are literally powering the world's economy. And so I think about data centers that started off occupying closets, growing to whole floors, then buildings, then warehouses. And I start to think of the cloud as a building, as a factory where a bit comes in, gets massaged, gets put together in the right way, and then it gets sent out to its destination. And it really brings the internet home for me. This connects to one of my reflections, which is threaded through the invisibility of this infrastructure and what makes data centers and the connecting internet almost the perfect abstracted infrastructure. And what I mean by that is you can access them, you can use them, you can get all of the power of them without ever seeing them or touching them. I mean, now with us now living and mostly wirelessly blanketed internet ourselves, you have access to all of these warehouses and all these buildings via all these undersea cables. And if you're an engineer, you know, developing software, you can deploy to all these regions around the world and never actually look at ACPU or hard drive in your life. Yeah. It is, I think of physical infrastructure that humanity has built the best abstracted. Like it is physical, it is silicon, it is electricity, it is fiber, there is no real magic, it is all physics. But the only infrastructure I can think of in humanity that is as well abstracted is maybe money, but money is actually not physical anymore in the same way. Like this actually is still doing physical work and that struck me. It does, it feels very rare to have global infrastructure that is defining our life and how the world operates to be so invisible and so frequently in our life, even though it is invisible, that our entire day is mediated by this infrastructure. But if your entire day is spent on the train, you're very aware of where the tracks are and you can see them and you're on the train that's the engine. Right, even fire, we probably used it selectively throughout the day. The railroads we got on and off, the car we got in and out of, electricity, we turned the lights on and off. Hmm, this is a modern marvel that we're only not using when we sleep. And even then it's doing stuff for us. It's tracking my sleep. I'm wearing a ring that's tracking my sleep. It's sending bits to, you know, Aura's servers. Yeah, as you were saying that, it's like you're getting on and off a train. You're kind of like turning electricity on and off in a kind of transactional moment. This is almost more like you're living in it, right? You're living in this infrastructure, but we never see it. I just had a flash of Ready Player One. Yes. Putting on the full suit and immersing myself into another reality. We're close when you think about it this way. I think that's right. I think this gets to one of my other realizations when comparing this to other physical infrastructure, which is despite it being truly physical buildings and silicon and electricity, we are upgrading it continuously and rapidly. Hard of that is because of Moore's law. Hard of that is because of just the improvement of the hardware and our techniques and cooling and all these things. But I feel like most other infrastructure in my head, you get to, okay, this is a reasonably good way to build a train track. Now we need to go build more of it or this is a reasonable good way to transport electricity. So now we're gonna build more transmission. And certainly, you get a little bit better at transmission. We don't get a little bit better here. We're getting 40X better in the last decade, which was off of the previous decade. It's gotten much better. So this is weird thing where yes, it's infrastructure, but it is changing what it is we're getting at a such a kind of order of magnitude scale. So quickly, and I can't think of other analogies to that. Maybe the first 20, 30 years of electricity felt like this, but something feels even more drastic here because what is being delivered is not commoditized exactly the same way as electricity. - You know, that sense of movement that you're describing reminds me of another one of my takeaways, which is where I'm feeling stuck. And that's on the question of power. And to me, it's easy to think about the complexities of why the power problem is so hard to solve. And yet, the answers are right in front of us. The problem is huge, right? Data center is pulling more and more electricity from the grid. And the problem is hard because the grid is strained that the turbines and transformers are backlogged. And we don't really have utilities that are built to capture the incentive structures to value what hyperscalers are willing to pay today. And it's going to take years to reform the grid to actually meet the needs. And at the same time, the fastest cheapest energy is available today. It's solar, it's storage, it's putting renewable energy online quickly and cheaply. That is already accounting for 90% of new capacity going on the grid. Despite every effort from our current administration to execute an ideological war on clean power, and claim that we're in some national energy emergency, and yet deny the grid of having affordable electrons being placed on it. So if we can just get out of our own way, we can put a lot of power on the grid fairly quickly. The other side of that story is the way we're building data centers, we're doing it accounting for 24/7 peak load. And there's a famous study that circulated earlier this year from Duke University that suggested that if we can limit grid facing power of data centers by just 1% a year, 90 hours out of the entire year, we can unlock 100 gigawatts of load. We can unlock the equivalent of two nuclear fleets on the US energy grid. Right, you saw Google do some recent announcements with their collaborative with some of the utility providers to do exactly that. There's new startups that we see all the time that are building businesses around that concept. This connects really closely to my major takeaway on the power story, which is that limitations drive innovation, especially when they come up against a legacy and incumbent system, something that generally works and reasonably well, which for all of our general discrepancy about utilities in our life, it is phenomenal that the electricity works most of the time, as well as it does. And it has not gotten a kick in the behind in a long time in terms of needing to try and evolve. There's been decades of people talking about a smarter grid, but nothing's forced the need, like the AI data center competition and the geopolitical competition with China, that drives innovation across the stack. It's exactly what you're saying in terms of driving the next cheap clean electron on the grid in the forms of renewable energy, but it's also pushing us on transmission and permitting. And it's also pushing us inside the data center to think even further about AI models. And we look at an invested in companies that improve the efficiency of actually running the model and training the model. That's right. And so up and down this whole stack, a point that we found again from talking to people in research, these types of crunches drive the efficiency innovation and drive the evolution of the grid. And arguably, according to folks that have been in this for decades, we've been fairly lazy on innovation. It's been a lot of the low hanging fruit. And so this type of forcing function can drive step change improvements in something is complicated as our national grid. So to me, all of that points to, for me, long-term optimism in, first of all, solving the problems, but second of all, solving them in a way that leads to more clean energy deployment, but also needing to recognize and not dismiss the near-term local impacts for both power and water [BLANK_AUDIO] in these environments. - I love that point because as we talked through these stories, that team kept recurring of how it is different when you think about the problem locally than when you think about the problem nationally. And I think that's really important for people to understand as this is more and more headline news. - My personal takeaway is less and less anxiety about that next incremental Gemini query, chat, LGBT and thrall pick use case or story video. And I'm more desired to have more awareness locally within a given county that's struggling with air quality or water shortages because that's where the activism needs to center within communities in the moment then trying to slow the valuable uses and progress that we're all building towards. - Absolutely. This point on communities and geography and where things are located, brings up another takeaway that's close to my heart as a former international politics major and someone who spent 20 years living and working across emerging markets. And that's the geopolitical implications of the moment we're living through right now. As we talked a little bit about this, but to me it really crystallizes into twofold. One is the notion that we're living through this chip war, right, that microchips have replaced oil as one of the world's most critical resources. And it's a key determinant of economic and military power and we're in a new Cold War this time it's largely with China. As America's we don't want a country like China accessing our data or having superior performance over us. And so we're actively preventing U.S. companies from selling chips to China. But it's not so straightforward right, because you think of okay, Nvidia is an American company, we can prevent that, but Nvidia's the architect of the chips, right, the manufacturer of over 90% of the world's advanced chips is the Taiwanese company TSMC and they're the ones actually constructing it. And so this is geopolitical tension has various layers to it, but the notion that we're living through a new Cold War and the chip is the center of that story is quite striking. I think in the long term or mid to long term, this type of technology is a genie that can't be put back into the bottle, right? And that China in particular is hyper aware of what they wanted domestically built for themselves here. And actually isn't a great position from a supply chain perspective to pull that off, even if they're a couple years behind. You've seen them spin up the AI labs, spin up their own domestic chip manufacturing and do impressive jobs. And you can say, well, where are you team months ahead? But what is that in the scale of this kind of foundational technology? - Yeah, 18 months is a blink of an eye. - I lean again towards long-term optimism that this competition creates good incentives for us to rethink our fundamental constraints, which are mostly on the clean energy deployment side. One big thing China has going for it that we don't have is their ability to grow generation of clean energy much, much faster than we do in the US. Again, because of their solar panel supply chain, their battery supply chain. - Their government structure, that their ability to build is unmatched. And so this is giving a lot of our structures that kick to rethink what's slowing us down. If this is so accidentally important to say, at least on par if not ahead on. - I agree strongly. In this notion of the chip war, I think it's interesting in this moment, I think over the medium to long-term, it's less so. Here's where I don't share as much optimism. It's the AI boom and data center build out that's potentially creating a new digital divide, exacerbating global inequality on yet another dimension. So we've talked about how US hyperscalers run almost 50% of data center compute. The US China and the European Union host more than half of all the powerful data centers. What does that mean for the billions of other people living around the world? And what is the implication of a country who is renting compute power from a far away data center? It's potentially higher costs, slower connection speeds, compliance to different laws. But more importantly, as we unpack this, as we're trying to wrestle with this idea, it's this question of having a critical national security and economic infrastructure input owned, operated, and housed outside of your control. - The techno optimist in me is just screaming to like, "refute this," and I'll tell you why. And I think we can study the internet as an example, which is like, this feeds into, I think, another meta theme for us, which is these companies are the modern day utilities, but they're actually the opposite in a way of the governing structure of electricity utility, which is hyper-localizing, compared to the local monopoly. These are global utilities. They're providing much more kind of non-commodity functionality, everything down to a, you spoke AI model that they're training down to the kind of more commodity storage. And the business model of utility is too, as we've seen again and again, provide more as cheaply as possible to as many customers as possible. And you've seen the biggest internet companies grow by incentivizing and subsidizing free services to more and more people. And that has led to a lot of the world getting connected, that was not connected just 10 years ago. There's reason to be cynical about what is Google and meta doing trying to get more of the world connected. Don't they just want more customers? Obviously, yes, they want more customers. But in doing so, a lot more of the world has access to the internet than ever before, and information and knowledge. And I think that's where I fall on the side of optimism that there's nothing more that Open AI wants than more people having access to Open AI. Now, that to me goes to the individual human level, very different than how a nation state or the government in control of a nation state might feel about it. But I lean on the like individual human in a global context and what powers I want them to have and I want their children to have in the decades to come. And that's where I get a source of optimism. I mean, it's true, 66 plus percent of the world's population is now connected to the internet and that number is increasing. I like the optimism. I wonder if the nature of this technology, particularly AI compute power, functions differently than access to the internet, access to electricity and access to other similar infrastructures, having that controlled by private equity and public companies that have very specific incentive structures. I think there's a big thing that you just hit on for me, which is my optimism doesn't really count for the hosting government of paper scalers. And so it's not just that these are companies, but they're American companies. If they're American companies, they a little bit exist at the pleasure of the American government. And that means that another country to access to them exists at the pleasure of the American government. And that I think gets to a more complex nuance. And to me, just drives value to the idea of more global, models and more global competition, even if it's not hosted within a given country, so that it's not just one or two governments that have control of these technologies. - More models trained on different languages. - Of course. - And different speech patterns. And so how this unregulated utility, as we're talking about, this new digital utility, this new critical infrastructure that society will increasingly depend on, will continue to be seen as a public good and delivered to more and more people as that. You know, they say hindsight is 2020, but there's something even more interesting about creating a historical narrative to something like data centers. And it's the marvel of the narrative process, the story that starts to make so much sense of how one thing came after another. And so you start off with IBM and Watson's journey and the introduction of the punch card that is storing data. And then the machines that start to compute off that storage. And that kind of grows and evolves and you've got the Cold War introducing the need for connecting networks. And suddenly you've got compute and storage power and network connections, bring that together. And that continues to evolve and then you get the explosion of the '90s. And in this time, first slightly accidental and then more purposeful infrastructure starts to get built in very important nodes. And the fiber boom happens. I think I'll now always associate the late '90s with the over-build of fiber in this notion that by the time the dot-com bust happened, we only use 3% of the fiber that we built. - Wow. - And yet that was exactly what we needed to go through the run-up of the 2000s and the growth of consumer and enterprise internet and mobile and streaming and then eventually the cloud. - And then you had crypto creating this market for the AI chips before we had the models, cycles of building ahead of the curve. There being some bust, but then there being this new utilization of those fundamental building blocks. - Absolutely. And these brilliant people located in very important companies that took the lead to build what is now foundational infrastructure of the world. And then there's another dance that's happening in where storage and compute lives for the average consumer or average business. Where you go from, it was all on the main frame. Then it moves eventually to the PC and you feel like you're so powerful with your PC but you have no collaboration, no backup, no redundancy. Then it moves back to the server room. Then in the mobile area, your connection is really. slow early days, right? You think about the first iPhone. It was like a really slow edge connection. So your phone actually had to do most of the stuff on the phone. But then now we live in this world where the connections back fat and happy. And so like we do everything now on the server, except for, you know, rendering the UI on the phone. And so this dance of where both compute and storage happens, it mediated by the communication bandwidth. This really interesting dance that I see played out in the story. That dance drove then the cloud boom of the 2010s and the maturing of enterprise cloud and the infrastructure where now you're able to do everything you want at higher, higher utilization in the cloud. And all of that build out and that fine tuning and optimization allowed us to enter March of 2020. The dark days of the beginning of the pandemic and switch over from in person to online from the perspective of the internet, largely without much of a hiccup. Yeah. Going to slightly lower resolution Netflix videos for a month, I think we handled it pretty well from a technology infrastructure perspective. Correct. And how that just so happened to juice the infrastructure even more to lay the foundation for GPUs to finally be utilized for what was growing in the background around machine learning and large language model training. And then November of 2022 putting a chat box on top of a model and the AI boom and the infrastructure that was laid to capture that moment and the cash coffers of the hyperscalers that were ready to meet that moment that are now propelling the greatest technological infrastructure build out in human history. What a story. Yeah. Do you think of an overbilled? There are things that I worry about in this world as we noted from the geopolitical situation. But once I strap in, the optimist in me takes over. So I am fully convinced that we will utilize every photon, electron, and square footage. I hope that this is the opportunity where we collectively can seize the moment to get more clean firm power online or affordable power online where we can use technology and renewable power to reduce the depletion of water resources in critical areas. The opportunity to build a newly skilled workforce to build out this new infrastructure. The opportunity to modernize our aging grid, our transmission lines, the way we do citing and permitting, the way we do environmental review processes, the way utilities are incentivized to meet demand. The opportunity to spur local economic development in communities all over the country in the world. The opportunity to have large enterprises build out infrastructure that will be a net benefit to the consumer where they're locating their assets. And I think it's really important for us to be mindful of the cost of moving so quickly and not thinking through the implications of water, power, climate, communities, and global equity. And my real hope is when you have this much capital and this much demand coming on to reshape a foundational technological infrastructure that we come out the other end with an ability to, as one of our new friends that we interviewed said, to build these assets as ecologically invisible as possible. I think we have the opportunity to do that. And when we come out the other end having done that, I think our show and generation will look more kindly on us. I think so. My big takeaway is that I think of all the things we might study on the show. This one feels like the major piece of infrastructure that we're living in the middle of the step change. We are in the sharp part of the curve that's going up. This level of capital investment and build out is what it looked like if you were to put yourself back into the initial build out of the electricity grid or the railroad system, but further compressed. And I think what's so interesting about that is you get to experience personally how quickly you adapt to amazing infrastructure and power. How quickly we've adapted, I think, to having the latest, large language models at our fingertips. And now I have cloud code and I'm a few weeks into using that. And I have what feels like new powers, but I'm kind of getting used to it again. Right. And so we lived through most of this story and have vivid memories. Like I remember blockbuster. And I remember when Netflix was a DVD rental service. And that was because it was too expensive to move streaming video at that scale. And now it's blindly cheap. And so the business model changes the products completely change. And so we are living through this cycle of infrastructure iteration and evolution at a pace that's rapidly changing what we can do with it. And so I don't know again, when we get to the upper step of that curve, but we are living in the step change. And I'm just trying to relish that experience. The rocket ship is in take off. Feelin some ge forces. That's right. But you get used to it, don't you? Very quickly, shockingly fast. I mean, even since we launched the fund two years ago to now, yeah, our workflows have changed dramatically. That's right. Despite the fact that you and I continued to look each other 12 hours a day in the same screen. Yeah. Yeah. The normalization of humans is astounding. And then you start thinking about our kids and how they're growing up in this moment, speaking to Chattachy B. T. Santa Claus if they want or making a game. I mean, I've four year old programed a game that runs in JavaScript by telling Claude what he wanted. Amazing. It is wild, wild times. We are living through a step change moment. Thanks for listening. Hopefully. You're that much more informed about the invisible infrastructure that's not so invisible and powering the modern world. You know how you got this podcast into your ears. It's amazing. That's right. If you enjoyed this and have friends or colleagues that you think may find the story of data centers valuable, please send it their way. And please make sure you're subscribed to step change in your podcast player of choice. So you get to find out when our next episode comes out, which at this current pace will be at least once more this year. I hope. And send it for emails from us at stepchange.show. Podcast ratings make a huge difference to help people discover this. So we always appreciate it. And we love sharing from listeners, shoot us an email anytime. It will bounce from your data centers to ours over at high at stepchange.show. At step change, we invest in early stage companies accelerating energy abundance and building critical infrastructure. So if you're a founder working on software to help make all of this work from data center management to efficiency to power generation, reach out to us. We'd love to chat and learn more. And last but not least, we talk to a number of folks in the data center in cloud worlds who's helped us tremendously with this research. Deep appreciation and gratitude to Christian Velady, Peter Gross, Brian Janice, Sean James. Byron Rikitsis, Brandon Middell, Nat Bullard, John Coomey, and the thank you to Ben Gilbert and David Rosenthal for all of their acquired episodes on AWS, Microsoft, Google, and Nvidia. And a big thank you to Nick Patri, our editor. And to our wives, she Ben Anna, we're here to talk about data centers for the last six months. And all right, until next time, thank you. [Music]