Solidigm on Satiating AI’s Data-Storage Needs

Hello and welcome to the Tech Disruptors podcast hosted by Bloomberg Intelligence. In this podcast series, we speak with company executives and management teams about their views on disruption and how it is driving their decision making and strategy. Bloomberg Intelligence is Bloomberg's research arm and covers over 2,000 companies globally across multiple asset classes, backed by Bloomberg and third party data supported by nearly 500 research professionals. My name is Jake Silverman, technology analyst at Bloomberg Intelligence. And here today I have Greg Matson, SVP and head of products and marketing. Greg, you're here from solidime. I wanted to give you the opportunity to introduce yourself a little bit. So yeah, take it away. Hi, Jake, and thank you for having me. I'm excited that you're excited about storage and wanting to learn more about our role in AI today. I've been in the memory and storage industry for about 25 years now as part of Intel, their flash memory business, and then in 2021, SK Heinex actually purchased that division and created a new US subsidiary called solidime. And since that time, I've been in my role in helping us shape and grow our data center strategy in our SSD strategy ever since. Great. How long would you say you've been at that solidime? Well, since the beginning, so that was, you know, solidime itself has been in existence just a one month under four years. And then the flash memory business with Intel started in 2006. And then before that, we were actually in nor flash memory, which was mostly used in consumer devices and mobile phones. And that was even from the 90s. I started exactly almost 25 years ago. So you've seen all the cycles seen all the cycles and there's it's been a wild ride. It's one of the things I actually like about the industry. Now it's maybe not always the best thing for my, you know, pocketbook, you know, and the down cycles versus the up cycles. But, you know, it really makes it a challenging business to be in and mastering that challenge is kind of what I've what I've grown to like. Yeah, it's been quite the evolution over the course of the history of NAND and obviously nor going back far as well. So I did want to take a little bit of time just to explain a little bit of the, the history of a of solidime. I know you introduced it a little bit before. But, you know, what was the, you know, ultimately the strategy behind, you know, Intel sort of selling that NAND business over to SK Heinex. And does SK Heinex sort of, you know, benefit from the enterprise side of solid, the fact that solidime has such a strong enterprise business. There's some synergies across capacity and supply areas like that. Yeah, for SK Heinex, I think it was a very logical acquisition to choose the solid state drive business of Intel. We were historically very, very strong in the enterprise or data center SSD market. SK Heinex was more focused towards mobile and consumer with just kind of an emerging SSD business. And so it really made a good combination between the two businesses in terms of complimentary market segments, not a lot of overlap. And, you know, market expansion for SK. And, you know, in addition, our technology roadmaps were different. They, they focused on charge trap NAND flash, which is a more performant power efficient type of NAND flash memory, not as good for high capacity, where we were focused on floating gate NAND flash memory, and which is ideally suited for high capacity, highly reliable power efficient drives. And so to give listeners a little bit of a background, solidime right now, according to third-party researchers has about 20% share of the data center market. So definitely one of the leading suppliers. And, you know, going off of what you just kind of said, you know, solidime and SK Heinex, there's different product roadmaps. But if I understand correctly, at some point there might be a convergence in the future, you know, what is the key role that solidime really plays in that? I know you touched on a little bit, but maybe there's some areas you can expand on within that. Yeah, there already is. We already achieved some synergies, right, where we've adopted some SK technologies, specifically their charge trap NAND flash based products, and then adapted them to our customer use cases. So we customized the firmware, the form factors, and in fact, you know, we launched the world's first liquid-cooled SSD, and that was a solidime innovation in terms of thermal management and system innovation on top of SK technology. And, you know, on the high capacity flash side we've been sticking mostly with, you know, the historical, you know, floating-gate based products that we've had and built over the past decade. We've also been able to take the, we're very good at customer scaling. We work with most customers, most big data center customers, and many, many small ones across the globe. And where SK's footprint in the enterprise space was very focused on their high-volume DRAM customers, and where they wanted to be kind of a full-stack partner to those customers. Yeah, it makes a lot of sense. And, you know, solidime now really, primarily, as you mentioned, enterprise data center focus, and the company made the decision not too long ago, but a decent amount of time away now to pivot from a PC market and really focus energy on those types of customers we've been talking about. You know, what went into that decision? It seems somewhat, you know, precious now, given there's so much exuberance around AI. It's a very strong NAND cycle right now, and I promise we'll get into AI in a second, but maybe we could start there. Yeah, it actually started. The most visible version of it was in 2023, kind of early, where we decided to focus all of our resources on enterprise SSDs. We saw the market fragmenting and changing very, very rapidly, and we were also getting a better return on the investment for R&D from that perspective. And so we decided that we wanted to differentiate ourselves, you know, compared to all of our NAND competitors, no one else focuses only enterprise. We wanted to be the best in enterprise. We had actually started down that path maybe a decade before, by focusing our PC based business on just business client for the most part. We had a few consumer products, but not a lot. And still focus most of our business on enterprise, but we really doubled out on that in the early 23 timeframe. And shifting gears to AI, because it's such a hot topic right now. SSDs for AI, there's been a lot of chatter about inferencing's need for more SSDs, higher capacity drives. Maybe we could talk a little bit about what's behind that trend. Is it mainly models storing user data to improve experiences and performance? And how does this differ from what we saw with AI training? Because while that was certainly a tailwind from the outside looking in seemed fairly modest. And then, you know, I guess we can talk a little bit about AI video creation, but let's start with that. You know, first maybe it's good to understand the two main places where flash memory is used in AI. One is inside the GPU server itself. And where today's reference designs from Nvidia, for example, have two SSDs per GPU inside that system. And last year those SSDs were like four terabyte SSDs this year. They're eight terabyte next year. And you know, as kind of the GB 300 time frame, they're going to be like 15 terabyte drives. So in, you know, each drive is getting bigger. Inside the box. Now, those are very high performance drives. They're really to help, you know, do a couple of main main things. One is to have near access to a large amounts of data to the GPUs, right? Obviously, there's limited amount of DRAM, limited amount of HBM. And they need a pool of memory that is much bigger, you know, close to them. Some of the things driving that are like within inference itself is, you know, the KV cache, for example, is the kind of the system's short-term memory. And when you're answering when you yourself just go to chat GPT, ask a question, get an answer. The GPU has to remember this, right? Then you ask a following question, a following question. And pretty soon you have a pretty big contact size of data that you're in that state that you're saving. And that's just kind of for a chat-based thing. Well, think about if you're doing, you know, large inference run, you need to check pointing. So the other thing that these SSDs inside the box are used for is check pointing and needing to not lose your work along the way. And as those jobs get bigger, you know, whether it's through text is kind of a little bit small, but as you get into multi-modal stuff, then you need to check point this big data. And it's just, it's basically like a snowball effect of how big the data gets. The other place where SSDs are used is in network attached. And much larger, maybe three to four times the size of what's inside the box is outside the box, network attached. And that's to feed, you know, again, it's one more tier of feeding, you know, massive amounts of data to the GPU server. Yeah. And then speaking of that, AI video creation, we've seen a lot of interest in that area. Sora, I think, has garnered a lot of media attention. All these AI-generated videos, the quality continues to increase. Do you think that's a long-term catalyst for storage? Have any of your discussions with customers in terms of demand changed because of some of these multimodal types of AI? Or how do you think about AI video long-term or multimodal generally outside of just sort of LLMs? Well, in terms of the use trends, it's hard for me as a storage guy to totally say, but I can say that, you know, the video type use cases and multimodal, you know, the output of those can be like thousands of times, very 10,000X the size of a text-based LLM. So it's driving a massive amount of storage. You know, for us, it's a massive amount of storage, it's fast storage. And, you know, just a, I'd say probably a pretty strong long-term growth opportunity for us. Now, you know, how does that evolve, you know, at the use case level, you can only imagine that, you know, it's things in the last two years since this has all started, moved at rocket speed and it's probably going to go even faster. Yeah, and something we're not really accustomed to hearing about shifting it a little bit, but staying in the same AI land, we haven't really heard too much in the past about near-line SSDs. And for those familiar with the market, and I'm sure Greg, you could explain this better than I can. But, you know, near-line is typically something we see with hardest drives, right? HDDs and cloud storage. There's an HDD shortage right now, and hyper-scale customers are, you know, resorting to near-line SSDs in response. Maybe talk to me a little bit about that, you know, where does solid-eye and some of your peers in the space on the SSD side? Where do you play a role in this? Do you think, and then plan off that, how do you think about potential sharegames? I mean, you know, as we move over to, you know, continued vertical stacking, you shift over from trial-level cell to quad-level cell, you know, how does that impact the economics of NAND versus HDDs? At some point, do you think as HDD capacity increases, do you think some of those shared dynamics could revert back and regain some of the lost share? Well, I'll start by answering your last question first, and that is, no, that I do not see, I think this is a one-way shift in a very tectonic type scale of a shift of moving from storing data on hard drives to storing data on solid state drives. And the reason is, is because all this data is becoming, you know, the whole data hierarchy, you just think of a pyramid, right? And hot data is always at the top, and it's the smallest amount of data. Well, with these AI models, the more data you have, the better the model. The more you do inferencing on that more data is you're creating even more data. And so all this data is becoming warmer, some of the warm data is becoming hotter, and just the performance attributes of hard drives cannot keep up. You know, their technology roadmaps are slowing down in terms of a capacity per drive perspective. And also, as those drives get bigger, they get slower. And so it's just really not a match for modern AI infrastructure, whereas by adopting high capacity solid state drives, you can, you can save as much as like a nine to one storage footprint in the data center, as much as 90% of the storage-related power. And as you know, anyone building a data center wants to fill that thing full of GPUs, because doing inference on GPUs is how they make money. So they don't want power. They don't want the space taken up by storage. And so by adopting the most high performance, high capacity storage, it actually saves them power, save them space. And it's changing the economics of the TCO crossover. Whereas historically, there was always this magic 3-to-1 ratio. We could be within three times the price of a hard drive, it totally makes sense. That is actually changing based on the performance needs and the other constraints, power and space, making flash a much more attractive purchase within your AI infrastructure. So once that's made, it's never going back. Now, will hard drives be around forever as long as I can see? Yes. So I'm not going to say that I've heard people say, "Oh, there's going to be, you know, hard drives are going to be designed out by, you know, 2020-28. I don't think so. But I think that the very large percentage of hard drives will be taken over, or I would need to say hard drives taken over, of AI storage growth will be on SSDs. And hard drives will be moved to the, you know, very coldest of kind of data storage applications. Yeah, maybe breakdowns sort of hot and cold storage a little bit, where more NAND sort of differs in the data center from HDDs. Data is worthless if it's cold. You know, it can't be acted upon. AI needs, you know, it purely acts on data. That's all it does. And, you know, so SSDs are really the only way to have worm or even hot storage within the data center. And so a percentage-wise, if you look backwards, you know, maybe somewhere between 10 and 20 percent of the data was even stored, let alone acted upon. And now we're, you know, we're storing much more of the data and we're acting on much more of the data because, you know, it becomes now valuable with AI, you can actually create value out of this data. So you'd say, maybe you'd agree that NAND SSDs becoming increasingly prominent within the data center. Do you think it's going to increase, you've talked about a couple of different ways that capacity is increasing, right? We've talked about within the actual server rack itself, but we're also talking about, you know, network attached. I mean, do you think that this changes the way that data centers are sort of designed to an extent to allow for more storage capacity? Do you see any shifting there in terms of the way they think about networking architecture, anything like that? Well, I'm not sure I'm the networking, the definitely the storage needs are evolving in terms of on one vector performance is becoming super critical. And if you look back to like a general x86 box, they kind of used, you know, SSDs, the SSDs did compute functions, I did storage functions, and in some hyper scalers, they were kind of, you know, almost one size fits all, not all of them, but you know, kind of a SSDs and SSD, AI servers need max performance, and that's going to keep going. Nvidia is driving us to, you know, every generation faster, faster, faster. On the flip side is, and kind of the opposite direction is high capacity. The higher capacity we can get, the more effective from the power and space efficiency, you know, our storage becomes. And so, you know, we're kind of going in the opposite direction very, very rapidly. And, you know, so you have to be nimble and planned for those two trends. And speaking of sort of these maybe more industry trends, right? We've seen increasing capacity counts, right? We've seen it shift to 60 plus 120 plus, I think, you know, solid items talked about 245 or something around there. Yeah, are there any trade-offs between moving to these higher capacity drives? Do we, at some point, do we hit a sort of limitation in terms of how much capacity we can really add or you think AI is just so data hungry that we can just, you know, keep, you know, increasing the capacity or the demand for more capacity per drive will increase. We haven't seen a slowdown yet. Like you talked about, you know, we were first to 60 terabyte in 2023. First to 122, we were, we'll be delivering 245 next year. And in production at customers shipping, not at PowerPoints, right? And I see even within the decade shipping a one petabyte SSD. And this is for sure on our technology room, right? Customers want it every day they ask. In fact, as SSDs get larger, they actually get more reliable because the failure mechanisms within SSD is actually not the NAND. It's the other stuff. And so the ratio of other stuff to NAND, you know, decreases, your failure rate actually goes down. So it's a kind of a people get worried about blast radius and failures of any individual drive. And certainly there's blast radius things to take care of, you know, within the software that manages the data on the SSDs. But it's manageable. And it'll be more reliable. And more power efficient, more space efficient. Definitely want to touch on power. And it's sometimes done a lot, right? In terms of innovation, you've talked about some of the leading products that you've had first market, clearly a very strong market share. You're supplying to most, I would assume, most major data centers, hyper-scalers, or data center operators, hyper-scalers, enterprises. You know, what is sawdime doing to differentiate on, you know, some areas like liquid cooling, right? Because we've seen micron, Samsung, they have similar offerings. Obviously, we're seeing efforts from companies like Sandisk trying to carve out their own slice of the market with high capacity drive. So I guess, you know, what is, how do liquid cooled SSDs fit into the picture? How is solidime positioning itself to differentiate from the competition? Yeah, first we differentiate by we actually invented the first one, right? We partnered with NVIDIA to even design the carrier and the mechanisms for inserting the drive into the system and getting the liquid to the SSD. Those were all co-invented technologies. And we're not going to stop inventing there, right? We're the only SSD vendor qualified on the GB 300 liquid cool platform. And, you know, the competition isn't there yet, right? Same thing on the high capacity, where we were we invented QLC, you know, in the data center for the most part, you know, we commercialized it first in modern NVME drives. We were first to the highest capacity points along the way. And going back to my, even my last comment is we were first to ship them reliably in very, very high volumes, right? Not first to put them out on a piece of paper. And so there's, you know, we have a, and we have future innovations coming, you know, in both the cooling side, across software innovations, we have, you know, in addition to the capacity. And we're, we're evolving also, you know, continuing to develop our nan flash technology to get to the next highest capacity point, you know, as well. Yeah, I mean, so our view a little bit is that some of these things, the innovations that you're creating, well, it certainly is really interesting. If someone becomes almost table stakes to a degree, I mean, do you think that it's just the fact that you guys have such solid relationships developed over many decades, plus the fact that you have such a time-to-market advantage for the most part? Do you think that gives you a stronger relationship with a lot of your customers? We do a lot with our customers. We've over the past 15 years, you know, in the data center SSD business have taken a great pride in our deep technical relationships with our customers, our understanding of how our drives fit into their systems, of what their software does to our SSDs, how to build reliable SSDs that when you plug them in, they actually work, and you can qualify them easily. And, you know, those are, those are non-tangible innovations that really gave us a lot of preference out there in the market as well as co-innovating with them. So the example of the liquid cooling is just one, the most recent one, but it's like we're co-invating with our customers, you know, doing, you know, with them solving their problems, right? And so if you look forward, you know, liquid cooling for the entire rack is going to be what's required from Nvidia, and storage was one of the last components and we helped our customers solve that problem. So I want to talk a little bit about the memory cycle. Yeah, there are a lot of demand drivers today. AI, I think, namely is the most prominent, most important, but as we talked about, there's HDD shortages. There's really an under-investment in capacity that's happened over the last several years. And that's partially because of, you know, the scars of the most recent up cycle, right? We want to look back at 2021, 2022. You know, our view is a positive pricing is probably, you know, sustainable over the next 12 months. I'm curious, you know, how do you view the market looking forward? Do you think the industry can remain disciplined on the supply and capacity side, you know, even if we end up with an under-supplied market for an extended amount of time? Yeah, I mean, first talk about demand because the demand signal, as you alluded to, has never been stronger ever. And we can see for sure, you know, the next two, three years, very, very solid demand. Let me kind of explain why that's very uncommon in the industry. Oh, yeah. Yeah. And the hard drive guys are telling you that too, right? They're, you know, 12 to 18 monthly time, right? Build order kind of thing. The driver is just the absolutely sheer magnitude of the A, the technology shift from hard drives to solid state drive, but also just the size of it. So we've calculated that for each gig a lot of data center capacity deployed in need about 550 KGPs. The GPUs have a very specific you know, set of math around how much storage gets deployed with them. And there's direct attached math that you can find in Nvidia's reference designs. And then there's network attached. And between those two, we think that for every gig a lot, it's about 25 exabytes of new flash creation. Now just look at this here, you know, that we just pick 12 gig a lot and level announcements of kind of the top biggest ones. Just 12, well, equal 30 gigawatts of total capacity, well, 30 times 25 and 750 exabytes. That's three times the time last year of the entire ESSD market. And that's new Tam between 26 and 2028, just three years. Well, to put that in context, there's, you know, let's just say that's around five typical size nann fabs could be plus or minus depending on the nann fab, that's massive amounts of debit data on top of the demand that we already had. Now, customers, you know, suppliers are already started to mix out of lower margin businesses, mobile PC, but that can't be sustainable or those industries suffer as well. And that port of the market is going to go up. And that's not really a problem that you have necessarily because your business is so strong in the data center, but it is part of that price equation, right? Oh, for sure. Yeah, prices are very healthy. You know, I mean, you know, our demand goes up like 10X in the past two demand cycles. You know, that's that's really strong demand. And then bodes well for the pricing environment, right? You know, and so, I mean, you can read about people increasing price and announcing publicly, we don't, we don't advertise those things, but, but we're, we're enjoying the benefits. Right. I mean, it is interesting because you are in such a slightly different position than some of your peers, but also it's just sort of an unprecedented situation in the industry having such extended visibility that is multiple years out. You're one of your peers, Sandisk, the exact same thing on their recent earnings call. And, you know, it's just something that gets a lot of investor questions that I get a lot is how long can we see the cycle? And it makes a lot of sense from Solidime's perspective that, you know, you want to be able to address this demand, but you also have to think about it from a sustainability point of view. And you want the healthy pricing because you want healthy margins, but you also have to think long term about, you know, your customers meeting their demand and not being left with excess capacity. Right. Right. Well, we're in an environment where customers are willing to make long term commitments. They can't ship GPUs without SSDs. And, you know, any, you know, analysts can go and look at all that the pipeline for GPU deals that have been announced, let alone the customer silicon stuff that's happening at the high-scalers and, you know, the small, you know, the kind of the startup environment. That's not even counted in what I'm talking about. And so I think from a long term perspective, you know, we have 10, 20 years to get mature. It's going to be up into the right the whole time. Now, is it going to be a linear line? Is it going to be a nice smooth line that's comfortable for everyone? Probably not. So, you know, it's going to be fits and starts. Especially because there's other constraints. There's power constraints. That's our biggest one today, you know, in this country, but even worldwide, space constraints. There's other types of silicon constraints, you know, and so it's not going to be, you know, we do have to be responsible with how we deploy our capital, but it's certainly an environment where that makes sense. Thank you, Greg, for joining us today. Thank you, Jacob. It's pleasure to be here. For our listeners, we want to thank you for tuning in. If you like the episode, please subscribe and leave a review and check back to your conversations with the leading disruptors in the tech landscape. If you want to learn more about our research, including deep dives and topics like AI and memory, check out our work, Terminal at BIGO. We also want to thank Lydia Somani for helping in editing this podcast. This is your host, Jake Silverman, signing off.