Speaker 1I would say I'm overall opposed to, you know, the pace of the frontier directions going in. The scariest thing to me on the political spectrum and the way all of this is being treated is that it's now become an almost unifying issue on left and right about being anti-data centers. I think that is almost entirely a Chinese psyop. A single in and out uses, you know, more water than, you know, the largest data centers in the United States.
Speaker 2This is 20VC with me, Harry Stebbings. Now, we have probably one of the most important times in history for technology. We have the biggest model providers saying we need to pace the frontier. But what does that actually mean in reality? How possible is it? What does it mean for the threat from China? What does it mean for the infrastructure layer moving forwards? We have a true expert of the space on the show today in the form of Thomas Somers. He's the co-founder and chairman of Positron AI. They just raised an $875 million Series C at a $5 billion valuation. They've got some of the best investors in the business, including the one and only Gavin Baker at Atreides. And many more great names. Thomas did not hold back in this episode. And it's this beautiful combination of incredible education on the infrastructure that powers this economy for AI. And then also, I don't know how to say it, but analytical gossip would be a more intellectual way of saying incredible discussion about what we can expect in the next few months from the biggest players in this space. But before we dive into the show today, the best model for your application might not exist yet. The most ambitious AI teams are training open minds. To beat the frontier in their domain. Now, on November the 3rd in San Francisco, Forge by Fireworks brings those teams together. People like Jensen Huang, CEO of NVIDIA. Michelle Kataster, president and head of AI at Replit. Lin Kuao, CEO of Fireworks and more. To hear how AI leaders are taking control of their differentiation, margins and roadmap by owning their intelligence. Learn how they're building it with inference, training and intelligent routing. Forge is free to attend. But space is limited. Apply today at fireworks.ai/forge. While Fireworks AI powers product intelligence, Asana keeps the work moving. Most companies have tried AI. Most aren't seeing results. Not because AI doesn't work. It's because AI hasn't reached the workflows yet. That's the gap Asana is built to close. Asana is the operating system for human agent teams. Your easy button for AI productivity across every team. Ready to go AI teammates. Prebuilt for marketing. Ops and IT. No prompt engineering. No setup. They show up where the work is happening. Already onboarded in your workflows. Ready to deliver. With Asana, your whole company can work on the same plan towards the same goal. Whether you're a team of 10 or a team of 10,000. Asana, where humans and agents workflow together. Try it at asana.com. That's A-S-A-N-A dot com. While Asana organizes the work, SuperHuman speeds up the day. Can I be honest with you for a second? Here's what my week actually looks like. Back-to-back meetings all day and the real pressure builds up in the gaps between them. I'll come out of six founder meetings with a whole stack of follow-ups waiting. And my inbox, it's hundreds of messages. And somewhere in there is the one thing that really matters, which I always tell my mother is her. I've tried AI tools for this, but they always lived in another tab. I'd have to stop, go find it, paste in the context, write the perfect prompt. It was just enough friction that it never really stuck. SuperHuman go. From the makers of Grammarly, totally different. It's right there where you need it, when you need it, in the doc, when I'm reading, the email I'm writing. When a thread gets dense, I highlighted and asked Go to pull out what matters without losing my place. And after a run of back-to-backs, I asked what was actually decided. And Go turns it into clear action items or drafts the reply I need to send. It's already up to speed, and I'm always the one in control. Try SuperHuman Go from the makers of Grammarly and find out more at superhuman.com. You have now arrived at your destination. Thomas, I am so excited for this dude. I said to you just before this that I think there are some big questions that the world doesn't know or things that they know that I think we're going to correct today. So thank you so much for joining me. Yeah, great to be here. Thanks, Harry. Can we just have a brief description of what Positron is and where does it sit in the stack?
Speaker 1Yeah, so Positron's a fabulous semiconductor startup that's building hardware, so really everything from the chips, the software, directly running on top of that, all the way up to the full systems and rack-scale deployments to power generative AI inference. So effectively everything in the hardware and sort of the low-level direct talking to hardware software stack that powers all of the applications everyone in the world is excited about right now. So everything from the likes of ChatGPT, Cloud, et cetera.
Speaker 2How does the infrastructure stack required for inference, what you're working on, change compared to training?
Speaker 1So training, I would say from the underlying compute level, fundamentally is a compute-bound problem. So it's a workload that's the more flops that you have. And if you look at, be it from the regulatory and some of the export control frameworks are heavily focused on the just flops required. So how many floating point operations per second can be done. And more or less, the amazing thing that the scaling laws of the past decade have shown is that the more parameters you add to a network and the more flops you dedicate to that during training, the better that that model is going to become. The big difference with inference and so the deployment of those models is the fact that for the actual math and the steps that you're doing is about half of what you're doing during training in terms of that the steps shouldn't be thought of as like the actual compute involved. But what it turns out to be is that that forward pass, that inference portion of it is heavily, heavily memory-bound due to the fact that basically for every single token that's generated, every little bit of output that requires going through the weights, the parameters, you could kind of, you know, from a biological sense, think of the neurons, you have to read the values of that for every single individual token. And so the way that that's, I think, very interesting about this, and I don't know if it says anything about the value or kind of actually saying that the inferences is somehow, I don't want to say more important because of course you have to train, but fundamentally when you're you already have the corpus, you already have all of the training data. And so with all that data, you can massively parallelize the token inputs, all of the sequences of words and sentences, paragraphs, et cetera, that are going into that. So that's something you can just crush through with a bunch of compute. But when you're inferring, because that's actually generated, you don't know what the token is five words down the line. And so you have to generate each and every one autoregressively or in order without, you know, foresight. And so that becomes a hugely memory bound problem that can't just be massively parallelized like training.
Speaker 2So I totally get that in terms of the shift from compute bound to memory bound. Is that what people mean when they say about the memory wall with regards to what you're doing?
Speaker 1Partially. I mean, the memory wall as a phrase has been around for a long time before, you know, the hype around AI. And really what it's come down to is if you look at the past 50, 60 years of computing, we've been able to, you know, have Moore's law giving us more. Transistors per, you know, square millimeter of silicon, you know, consistently. And while that has been able to result in, you know, greater raw computes, flops, et cetera, the improvement of the memory technology is not kept up at the same rate. So roughly speaking, you know, between, you know, 2014, just very early innings of the new AI era till 2024, you had about 120 X improvements in the flops of GPUs. So a single NVIDIA GPU had about 120 fold. You know, and that's what enabled a whole lot of the improvements over that decade, the improvement in memory band with there's only 17 X. So I would say like the, the real embodiment of this is we had massive improvements on a per device basis of the flops and then a whole bunch of elements on the periphery of improving the connectivity, et cetera, et cetera. But just like the ratio of the compute to memory bandwidth had this divergence. And so you had cases where if a problem was. Memory bound and you couldn't just scale the compute linearly with that, you were getting more and more memory bound as the decade progressed.
Speaker 2Why was there such a misalignment in the progression between the two ones, a hundred X, one 17 X, why is that the case?
Speaker 1Yeah, well, it comes down to a lot of, uh, technical implementation details. Like the fact that if you, if you look at the lowest level, the type of memory that is used on the silicon itself is called SRAM, static RAM, and SRAM is made out of, you know, six transistors with the. A bit line and word line, some other control logic around it, but that SRAM cell has not scaled in terms of the, the sizing of that with Moore's law over the past, about 15 years. So they have grown or shrunk, I should say, much slower than just a group of transistors that you'll use for, for other purposes. And I would say that there's just been a lot more architectural advancements that could happen on the compute side while an S 6 T SRAM more or less has not changed in. 30 or 40 years from like an architectural, you know, primitive perspective. And so, you know, that's on the input side of, raw technical capabilities on fabrication, et cetera, have not been able to improve, but I would also say that there wasn't the right motivations for most of that, that decade. So with convolutional neural networks, so things that powered like Alex nets, which, you know, really launched the deep learning revolution in 2012, and then res nets and, you know, all of the, I would say the, the advancements during the 2010s was in the realm. of machine learning models that were fundamentally compute bound. You could just throw more and more flops at CNNs and get better results. And you didn't really need all that much, be it memory capacity or memory bandwidth. But it was really with the transformer. And even though the attention is all you need paper came out in 2017, I would say it did not really get the attention, pun intended, it deserved until 2020. Well, GPT-1 and GPT-2 came out prior to that, 2018, 2019. But it was really GPT-3 showing that, OK, you go from a billion-ish parameter up to 175 billion parameters, and you actually get this massive improvement in capability. And that's really where I would say the transformer revolution started. And most people didn't really catch on to that until the end of 2022 when ChatGPT came out.
Speaker 2When you look at token economics and token efficiency today, what does no one know or talk about that you think should be much more front and center?
Speaker 1I do find it. Compared to a year or two ago, there are now different prices listed for cash versus uncashed tokens. But I don't think people realize how any providers that charge the same amount, even with a lot of people's cash prices, how high margin that is. It's like insane. You make all of your money on selling cashed input and output tokens. Why is that? Sorry, just so I understand that. Oh, because for like when we discussed earlier that both processing a cash token is, essentially free, it's one one thousandth of the cost, you know, order of magnitude of actually having to recompute and generate that token. So there is so much you can juice out of selling those cash tokens. And basically all the providers, they charge you to cash a token. They charge a higher rate than just, you know, the normal processing fee for like an input token. And then they charge you a lower rate when you read from that. And it's great when you're paying. That lower rate, but they're making obscene margin on that cash read. And there's a reason why, you know, Anthropic is being is reported to have, you know, 80 points of gross margin right now on API business. Were you surprised by that 80 points? Not really. I'm impressed by 80 points of margin and basically any industry. It's difficult to get that margin. And great thing about capitalism is those margins will compress with competition. So I'm confident and happy for that, even though those people are, you know, theoretically my customers. And my margin is sort of based on their margin. But I care more about, you know, a healthy ecosystem long term. It's more surprising to me how many people still today think that these are horribly unprofitable businesses and that the whole market is going to zero. It's absurd to me that the meme of open AI, Anthropic, et cetera, are just burning cash and eventually they'll run out of cash that they can burn. Like if they stop training, they'd be massively profitable overnight and there is a ton of other levers that they have without pacing the frontier. As Dario just had in his essay. I mean, I jokingly think that, you know, a little bit of the pacing the frontier discussion is, oh, this is a great way to reduce costs ahead of IPO. But I don't think Anthropic or anyone needs to do that. I think they're amazingly profitable businesses with their scaling rates.
Speaker 2And it would reduce costs just so I understand because they would spend less on training because they would be slowing down the speed. Yeah. Right.
Speaker 1Yeah. I don't think that's actually the intention or anything. But yes, that's the bulk.
Speaker 2How did you analyze the pacing the frontier? You brought it up. How did you analyze it?
Speaker 1I have mixed feelings on the safety topic. I am a human. I would like to live old age and more so than that have humanity continue to the stars and beyond. But I believe much more in the ability for this technology to revolutionize every part of humanity in a positive way. I do worry that pacing. In a lot of the ways that's being talked about, not necessarily how it will be implemented, has two big risks. One, a major pause and sort of that playing into, for lack of a better word, the blood eight sentiment that exists and that by pushing for pause, it's actually giving ammunition, giving better basis for those that actually just want to stop the technology over completely. And I see that as a major risk for humanity. The second piece is my actual piece. And I think that's what I'm most worried about in the world of technology. The thing that I'm most worried about of any AI outcomes is that technology and capability being concentrated to relatively few people and having a lot of what's being discussed from a regulatory framework and limitations on technology, et cetera, I think is like the modern road to serfdom. It's like the concentration of technological capability, making legal to do matrix multiplications is like the thing that will set us back to pre. Not just industrial revolution. It's like pre-enlightenment capabilities like that. That is like the biggest attack on classical, like liberal freedom concepts that I can think of, because while I do think that the vast majority of tokens are going to be produced by the big players, if the technology itself is restricted to just those, then they're going to be the new lords and kings and everyone else is back to serfs.
Speaker 2The greatest of respects is it not just lip service. Right. We'll stick meter in the corner. They can do that compliance and then we can IPO. Sam can have a reason not to IPO because his numbers aren't as good as anthropics. It plays into both our desires. And Elon wants time to catch up as well.
Speaker 1So it plays into everyone. Completely agree. And I think that is the biggest internal reason for everyone other than Dario. Dario and I would say the vast majority of people inthropic are true believers, both in all of the promise and capabilities. The technology and the risks. And if I were in their shoes, I would also be take it the massive amount of responsibility for that. But I will say, like, yeah, there's a lot of strategic reasons of saying, OK, by having these auditors, et cetera, that remove some potential responsibility, culpability from like legal perspectives, et cetera. The risk that I just don't think that any of I think this is a problem with a lot of very smart people, especially when they've amassed large wealth and power, et cetera, is they think that they're going to be able to keep that. The scariest thing and kind of my point, like a centralization of technology, like if it gets concentrated with companies, governments, et cetera, is you've got people that think that they're the smartest people in the room, not realize that they are not going to be the ones to actually control it when they put these measures in place. Like Dario, basically, on one hand, verbally begging for government governments to take over entropic. He's in part saying that because he doesn't think it will actually happen. I would love to see his reaction if and when that actually happens. And he realizes. Oh, shit. I thought that if I was begging for regulations, they would then make me the regulator. And when that doesn't happen and it just becomes a bureaucracy that halts all progress and the capabilities that currently exist basically get squandered to select bureaucrats, that's the worst outcome I can imagine.
Speaker 2When you consider the advancements that China are making, especially with their open ecosystem, that they are an incredibly talented ecosystem right now moving forwards. If we pace and they don't, what happens then?
Speaker 1I guess when I said that the worst possible outcome, I wasn't counting the Terminator outcome and I wasn't counting that. So I think, of course, everyone can agree Terminator or similar or similar is is very bad, but I think is extremely low probability. And and just I'm not a believer in that doom scenario for the vast majority of people would result in the same level of serfdom that I worry about with the scenario that I described would be. I mean, I think it would be significantly worse for some number of people in a Chinese CCP controlled, you know, super intelligent A.I. scenario. You know, on one hand, their strategic angle right now is have technology proliferate through open source, et cetera. I think as soon as they get into pole position, the latter gets pulled up with them in some way. I don't think they actually want the technology to be easily accessible to everyone now. I don't know if they will decide that. I don't think the world has some access to the technology, but they definitely will not let the billion people that are not CCP party members benefit equally from from technology.
Speaker 2So just so I understand, do you agree with it? Because to me, I just didn't get it. You can't pace the frontier unless the global A.I. community paces the frontier. And I don't see Putin signing up.
Speaker 1Agreed. And I think this is a little bit the same naivety that that I described by these company leaders and in general people in the Western world. I don't see Putin saying, oh, we're so great, we're so advanced, so far ahead that we can't get caught up to. I mean, on paper, is the U.S. the greatest military force in the world? Yes. If we had to all of a sudden have a drone incursion, the same level of what's happening in Ukraine, Russia, you know, coming up from Mexico. And if you take Mexico, just say that they developed, you know, very naive drone technology and et cetera, like on the level of what's happening in Russia, Ukraine and Iran. I don't see that as a country, if we had that coming up across our border, it doesn't matter our amazing military might. We built our military to fight the last war. And I think geopolitically our thinking is, oh, we're the big dog still. And that when it comes to A.I. technology, there is not the acceptance that export controls and all of the other elements that theoretically would allow us to pace and have people keep paces behind us just aren't good long term solutions. Do you think we should have export controls? I am very much a strong believer in free trade. and free exchange of ideas. The exception to that is a little bit, I think China has been a free rider of all of the benefits of a liberal free trade order for the rest of the world. Well, they get to keep everything closed off. I am very, very happy and think that any government societies, people that want to embrace free exchange of ideas and trade and everything else, we should have a very vibrant economy and ecosystem, but totalitarian regimes should not be able to participate with that, especially in the case where they get all of the benefits of that and get to export themselves, you know, things that make them better able to have that totalitarian system keep up. Can I ask, we mentioned, you know,
Speaker 2pacing the frontier and, you know, the different people who supported it. You had Zuck and Jensen say nothing. Well, Zuck actually come out in opposition to it, saying that we should continue as planned. What should we take from those two seemingly silenced and opposing it? I would say
Speaker 1based on... My overall beliefs right now, as probably evidenced by the conversation so far, I would say I'm overall opposed to, you know, the pacing the frontier direction it's going in. And so I appreciate anyone that is adding to the discussion that is, I think, being realist about the benefits and risks, but you always have to take that with a grain of salt of what are the motives of anyone that's discussing in it. And I would say I probably appreciate Zuck or Dario's conversation. I would say I probably appreciate Zuck or Dario's conversation. I would say I probably appreciate Zuck or Dario's conversation. I would say I probably appreciate Zuck or Dario's conversation. And I would say I probably appreciate Zuck or Dario's conversation. And I would say I probably appreciate Zuck or Dario's conversation.
Speaker 2You have no regulatory or policy restrictions. And I mean, the UK, you can't, you know, put up a paper aeroplane without getting a permit. So like, we're fucked. But you are getting there and you're getting, you're becoming a European state in terms of the regulation and policy requirements. Am I wrong? Am I being overly negative? I didn't get it.
Speaker 1You're right. I think the greatest advantage the US has in that regard is that there's still a lot of land, a lot of places that do not have all the same levels of restrictions. I don't agree with a lot of things of most, you know, administrations of my lifetime. But the current administration gets attacked for saying like, they're destroying our environments and, you know, destroying national parks, etc. The vast, vast majority, 90 plus percent, I don't know the exact numbers of national federal land is just open, empty desert in the West that is not part of a national park or anything. And the fact that there are so many restrictions to utilizing, you know, BLM land for building data centers, where it's like, you know, it's like, you know, it's like, you know, literally hundreds of miles from from any, you know, populated area. My great state of Nevada has plentiful geothermal, solar, all these green energy technologies, and we could, you know, build, you know, nuclear and other things in the middle of the desert where it won't impact anyone. And that there's restrictions to that is completely absurd to me. And I will say, there has been some political will and push to solve these things. But literally just in the past year, you have, you know, Republican governors and other politicians that at least had part of their platform to be pro growth and all these things backing away because they see from their own, you know, political base being anti data center based on completely false premises. And one of the points I want to go back to that you brought up was like, that people would have higher electricity costs, like this is the most basic supply and demand. If we increase generation capacity, and no one's saying we want to be taking energy from what's reserved for people's homes, like one of the regulatory problems I see is power companies have to have this offer of energy availability that is baked into the cost and capabilities for everyone. There's absolutely zero cases where a data center could be potentially pulling power from anything that's already been allocated. So that's just impossibility. And all these data centers that are getting built right now are coming with generation capacity that covers their own use and beyond that. And we're just not allowing them to hook up to the prices for everyone. And then you've got people on the power company side, like they're lobbying against new generation capacity, because that will actually, you know, market forces will more capacity will decrease prices, which would be good for consumers. So it's a very wonky market.
Speaker 2What percentage of data centers that are planned will be completed, do you think?
Speaker 1From the major providers, I think I would say that the capacity that they have planned, they may be in different locations. I mean, you've had some local communities that have successfully stopped facilities going in there. But then those data centers just move. I don't think a year ago, the major data center builders and operators were thinking that the political problems were as bad as they were. And so there is a lot more effort being put into education in those communities now, which I think will turn the tide a bit. But I mean, it's also just going to mean that those data centers move to locales that aren't going to have those problems as well. And I think that's going to be one of the things that we're going to have to look at in the future. So I'm not too worried that it's going to be like an existential threat and capacity build out. And then, of course, there's space if Elon's successful.
Speaker 2Do you believe that space is a viable alternative truly, or is it conference talk and lip service to justify a market cap?
Speaker 1I think something can start as one thing and turn into something else. I would never, ever bet against Elon. I mean, I primarily bet for Elon. If you asked me a year ago, I just would not bet against Elon. I would never, ever bet against Elon. I mean, I primarily would not have thought that there would be a good reason for it in the near term because it's going to be cheaper, easier, et cetera, to build on land. I also think there's great alternative technologies. A company we're partnered with, and I'm good friends with, the CEO is a company called Panthelossa that's building ocean-based data centers. Basically, a very interesting pumped hydro solution in the middle of the ocean. So there are alternatives that don't require going into space, I think, long-term. Part of the reason I'm a long-term big believer in space data centers is because I just think we're going to need to have a space economy for humanity to live up to
Speaker 2its long-term potential. Love that. Totally agree. I'll never bet against Elon. If we think about the cost of intelligence being tied to the cost of energy, how should we think about energy as a bottleneck moving forwards? To what extent is it? We mentioned policy and regulation being a core bottleneck. Is energy a bottleneck moving forwards or less than people consider?
Speaker 1I think there's two pieces to it. I mean, one, Positron is trying to deliver more computer capabilities per watt, per megawatt. And so sort of on our base case, we can turn what you would have spent 500 megawatts with NVIDIA equipment and do that in 100 megawatt. I don't think that's actually going to mean that you're only going to build 100 megawatt facility. You're still going to build the maximum amount of compute that you can. You're just getting more tokens, more intelligence per joule. And so if I go back to the long-term thinking, I think assuming humanity continues for thousands of years, everything turns into an energy problem. And you can go back thousands of years and just look at the progression of mankind. Fundamentally, that is a perfect track of our ability to produce and use energy, discovery of fire up to nuclear power plants. The simple tongue-in-cheek answer to your question is all progress is gated by energy. And even if there's energy available, it may not be economical. And so it won't be done. So actually, I would say the bigger limiter than just saying that energy, like our ability to build and produce energy, we've got plenty of technologies and capability to do it. say we've got way more economic limitations. It's like, how much debt is the world going to take on to build out everything over the next couple of years? It ties into energy, it ties into the infrastructure itself, et cetera. So I think economics is a much easier sort of scapegoat to
Speaker 2pick. I mean, people are already very concerned by the levels of debt being taken out in the debt cycle. Do you think their concerns are justified and do you share them? I think we've got a major
Speaker 1sovereign debt problem that masks a huge amount of second and third order elements in the financial system. Just the inflationary consequences of government that can print infinite amounts of its own currency. And the fact that we are, as we're already seeing the treasuries and the greater bond markets, that there is greater and greater perceived risk of the most quote unquote risk-free asset, I think will trickle down to all elements. So the financial system. And so when people worry about Oracle's debt and credit rating, I'm like, I believe in Oracle's business model and ability to execute and do everything a whole lot more than United States government. It's just the United States government can issue its own currency and also has guns and nukes to take tax revenue. So my biggest economic concern there is that there will be a more acute specific crisis that arises out of the compounding of national debt. Leading to devaluation of the currency that has all of the consequences downstream rather than like, I'm really not worried about any of the companies in the AI debt stream not hitting their revenue targets. Like if the past three, four years have shown we're accelerating every aspect of these businesses in terms of revenue profits and how they are improving the productivity and
Speaker 2value downstream. I'm jumping around, but fuck it. When I was doing the research, I was reading about chaos. I was like, I don't know what's going on. I don't know what's going on. I don't know what's going on. I don't know what's going on. I don't know what's going on. I don't know what's going on. And I was like, I don't know what's going on. I don't know what's going on. I don't know what's going on. And I was like, I don't know what's going on. I don't know what's going on. I don't know what's going on. And I was like, I don't know what's going on. I don't know what's going on. I don't know what's going on. And I was like, I don't know what's going on. I don't know what's going on. And I was like, I don't know what's going on. And I was like, I don't know what's going on. And I was like, I don't know what's going on. And I was like, I don't know what's going on.
Speaker 1And I was like, I don't know what's going on. And I was like, I don't know what's going on. And I was like, I don't know what's going on. And I was like, I don't know what's going on. And I was like, I don't know what's going on. And I was like, I don't know what's going on. And each of those little flashes that you see is a token. And roughly speaking, it's a token is equivalent to a half to like 75% of a word on average and large English corpuses. So that's token. A sequence or, you know, the thing that builds up to be in context in a model is the grouping of all those tokens in n order. And what happens when you're running an inference, you give it a prompt, you have, what is the capital of France as your input prompt, that is tokenized. You know, that is four or five, six tokens, and that that go into the model. And when you do inference, it's going to say the capital of France is Paris and the City of Lights, you know, some thing after that. And so when you have that entire sequence, when transformers originally came out, for every token that was generated, you were doing the computation for generating all of those tokens, including the ones that you've already processed. And, you know, clever, I would say kind of obvious based on, you know, all the development in the past of computer history, but wasn't done initially, was that, well, you don't actually have to redo the compute of the things that you've already, you know, had as inputs and what you've already generated in this turn. And so the KV cache was born where within the model, there's these two matrices called K and V keys and values. And those matrices are fully based on the prompt and whatever is generated during a turn. And so by actually storing those two matrices, you can avoid having to redo computation at the cost of now having to store this thing in memory. And that's, you know, the simple example, you know, very, very small, hundreds of kilobytes of data. But the thing is that these things grow with the sequence length. But the interesting thing is that for the attention mechanism, the compute per token grows quadratically with the sequence length. So you're having to spend more and more compute quadratically. So that's, you know, an exponential curve as sequence length grows. But when you store that as just your K and V, that's just a linear growth. What KV caching does is means that you don't have to do that compute, which gets very expensive very quickly at the expense of needing to store these things. Storing that is a complexity in itself, because that's a unique KV cache for every single user that you're serving. And it becomes questions of how long do you want to keep that for? And how do you manage all of that in a large system?
Speaker 2What does it mean then when we hear about compression and uncompression of KV caching
Speaker 1entropy within the system? So there's two different forms of compression, a couple more than that, but the two main ones. So one is quantization. So if you've got each of your values, your weights, your KV caches, activations stored in a particular data type. So before the machine learning revolution, you know, most of the world's computation was done in FP32. So you have 32 bits to represent a floating point number, and that's broken into mantissa and exponents. Well, it's pretty quickly realized that having 32 bits of precision was super overkill for the things that you're wanting to represent. And, you know, it's both cost more from a storage and computational perspective than lower precision. So we went to FP16, you know, Google developed BF16, you know, a little rejiggering of those bits, went to FP8, now we're at FP4, you know, in popular systems. And so we've been reducing the precision quite quickly, but that does lead to, you know, for lack of better words, some brain loss when these models run. Just because you are now trying to encode the same information into fewer bits. And so there's been a lot of interesting schemes to say, okay, I'm going to take this group of FP16 values, BF16 values, and I'm going to quantize those. So I'm going to, you know, use a truncate and rounding that down to, let's say, into four values. So now you actually saved 75% of your total size of that group of values, you shrunk that down to, let's say, 4. But just doing that naively will mean that on a lot of benchmark scores, you'll have them get 20, 30% worse. So you get that 75% savings in space, but you kind of lobotomize the model. But, you know, advanced quantization techniques actually say, okay, these 16 values, I'm able to have a, you know, shared bias and a multiplier for it. Let's say for those 16 now int4 or FP4 values, you would say, okay, these 16 values, I'm able to have a, you know, shared bias and a multiplier for it. So you get a 75% compression on all those values at the cost of now adding to add one new FP16. And basically the state of the art here is you're able to get things compressed from, you know, FP16, 16 bits per value down to like four and a half bits per value. And that can be applied to weights, the, you know, actual parameters and model that could be applied to the KV caches, but there's no such thing as free launch. So you can still have some lobotomy, but thankfully it's kept within like 1% of unquantized model.
Speaker 2Is KV caching the hardest element of building that inference infrastructure, or is it, you name it, latency SLOs or load spikes or anything else that we could come up with? Is that the hardest? Like, what do we not see that we should see?
Speaker 1You can run a service and do something without having KV caching at all. You're going to economics and performance and everything else. Why is it going to be? It's going to be much worse. The dark art and magic with it is the workloads that the industry so far has found the most valuable happen to be very, very highly cacheable. So semi analysis has their agent X benchmark and, you know, suite of test data based on taking a whole lot of quad code sessions and having dozens to hundreds of turns and those quad sessions with sub agents and everything else. And what they found is over these massive number of interactions of these, like, you know, real traced code generation agentic coding sessions, about 96% of all the tokens that go through these entire sessions are cached. So if you know your workload is going to have this extremely high caching rate where you're going to be reusing the same tokens again and again, that drastically shifts the importance of how you can retrieve those caches because these things get to be very, very large. Like we, you know, have gone into trillions of parameters. So if we just take, you know, the the GPT-4, you know, got leaked as, you know, 1.8 trillion parameter model. Now, assuming that that is in for quantized and rounding down a little bit, you know, that's 900 gigabytes of data size for the model weights. If we take like the high expectations of like Claude Fable, you know, that's a 10 trillion parameter model. So around five terabytes of model weights. But the crazy thing is at these long context links for these size models, you have the individual user sessions being in the, let's say, in the 100 gigabyte range. With just 50 users on your service, the user context, just those individual sessions end up being greater than the model weights that you're trying to store. So that's, you know, Claude and OpenAI have a whole lot more than 50 users. And so it becomes a really interesting trade-off of, okay, how much of the accelerator memory do you want to dedicate to weights, which you need to process every single token generated. And you want that to be as fast as possible because that sets your SLO. That's... That sets the token latency. But if you don't have their KV caches persistent, you're actually losing a huge amount of efficiency because that was work that you didn't have to actually repeat. So it saves you as an operator money more than anything. At some level, you know, having users KV caches be persistent will give some level of speed improvements that the user perceives. But it's mostly an economics thing for the service provider, where if you can return to them and use those tokens again and again, that saves you money as an operator massively.
Speaker 2Totally get that. It saves us money because we don't have to use as much compute, but then it's harder from a memory challenge perspective. How do you think about the right logical next step then? If you appreciate the importance of saving on compute, but the challenge of memory with KV caching, what's the answer then? That we just have bigger and bigger memory stacks on chip? What does that look like?
Speaker 1The most common deployed solution and, you know, the vast majority of inferences out there are taking place on GPUs, you know, followed up by TCPs. And a couple other devices. But the most common paradigm today is you've got your GPU accelerator memory that is primarily responsible for holding the weights. And you will keep some number of user sessions on there, the ones that you're actively processing. But the larger group of users has a tiered hierarchy. So you'll have users that were around, say, in the past couple of seconds, but haven't returned, don't have an active request. That's residing in host memory. And let's say that's on the order of anywhere from 4 to 10x. More memory on the host than in the accelerators. So you'll be able to store more of those there. And then if someone hasn't been around in a couple of minutes, maybe a couple hours, that's going to be stored in even further away memory. So that could be in NVMe, so flash storage, so a lot slower, but a lot larger capacity on that host. It could be in flash storage on a network-attached, you know, drive. And eventually, like, I bet the chat GPT sessions that I had, you know, six months ago, somewhere residing on a, you know, disk, you know, slow SSD or, you know, somewhere. In a data center. But it would be dumb for them to use expensive memory to store that. So that tiering is the norm. But that introduces a huge amount of complexity of how do you decide when and where you're going to store something for your massive number of users? I would say our solution, kind of how we're trying to go about it, you know, both from our expectation that model sizes are going to drastically increase, the number of users for all these things are going to drastically increase, and the context themselves. Like, two, three years ago, the typical context length. We're on the order of 8,000 to like 64,000 tokens that got up to 128, 256, you know, a million token context links are the norm now in terms of what the model support, but a million token context links can only hold, you know, a portion of some of like our internal companies, like largest code repositories, like it will be a fraction of that. And so if you really want an agent that can take over, you know, the capabilities of a whole team of programmers, I think the main limiter today isn't like the model capability. It's on how much context can that model have of all of the data it needs to make smart decisions.
Speaker 2I just want to break some of the things you set up there. You said that you think model sizes will increase. I thought we were all moving to owning our own intelligence, every enterprise having their own smaller model with proprietary data. Does that go against what you think in terms of model sizes increasing? Can you help me understand?
Speaker 1Yeah, I think you can kind of break it into two tiers again. So there's going to be the frontier models and capabilities that are being really at the forefront of the development by OpenAI, Anthropic, maybe Google, you know, SpaceX, AI, et cetera. I still think there is a long road to go in terms of getting to pushing the frontier of model capabilities, and those will continue to grow, continue to get better. And there are a lot of workloads where, let's say internal, just speaking for how Positron uses LLMs, where I don't today care that much about the cost. If I can get 10 times. The output value out of a model today, I very gladly pay 10 times more per token, and I really want the frontier to push that. Some of it is cost saving. Some of it is just truly owning your proprietary data. There is a push from enterprises to have inference on site, and it's a lot more difficult to provide that for largest models. And most companies, if they're adapting from open source or developing their own model, don't have the resources to pushing the frontier. And so that is kind of what's. It's like, oh, well, I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that.
Speaker 2I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that.
Speaker 1I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that.
Speaker 2I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that.
Speaker 1I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. I'm not going to be able to do that. on what you can generate but yeah gpt6 astra i do think is is agi and to to your question of like what does that mean going forward i think my guess is as good as basically anyone's but why was gpt
Speaker 2astra so good for you why was it comparably such a breakthrough because i haven't it's great but
Speaker 1honestly kind of the same as before oh um in terms of the things that i've found lms to fail the most out in the past so i'll give a case where it is more linear improvement so just in terms of general coding capabilities performance and analyzing problems etc it is a step function improvement but not mind-bogglingly so there are a bunch of things that other models have not been able to fix or kind of went in circles and found inelegant solutions and it's still like a human software architect that was able to come up with a better solution with astra they're just initially giving it a couple of really hard problems but it's still a step function improvement but it's still a problem that i've not been able to solve with other lms was able to do it one shot having a go through code base and find both performance improvements bugs etc basically discovering new spaces that i didn't know existed in our bunch of portions of our code base so that's one element step function but not mind-boggling the second case that was mind-boggling just from a like wow is the the computer use abilities with set of generic tools so like being able to do blender animations like you know it's become there's a bunch of memes online of of it recreating different videos etc but just the fidelity of that and where that was basically impossible with gpt 5.6 soul was massive increase capability and like i had it designed you know do interior design of my house just based on a couple pictures and just like wow like i did not think that what's fundamentally a text model could could do that and then finally like the the biggest thing for positron was i've been trying with every single new model release to have these models be able to like actually take a relatively simple logic design problem you know implementing a encryption block in this case and being able to take that through the full rtl to gds flow so from basically the specification of do this encryption function implement the verilog so the hardware description language for that so write that code and then be able to take that code and go through all the way until you have got a chip design that theoretically you could go tape out lms could do different portions of that and could like write the scripts and you know fail a lot of different midpoints on that so that's one of the things that i've been on the way but a big problem with the electronic design automation tools the eda tools for doing chip design is that they were designed in the 90s early 2000s they're really unintuitive none of the documentation exists out like in the public web and so that these models don't have like a real good innate view of them but gpt6 with both combination of computer use and just an ungodly amazing scripting ability has been able to take this this cat hack block and implement it the tsmc and three pdks and take that all the way to gds and do that in like a little over like 50 something hours and meet timing and you know over a gigahertz etc and that as a task if i was giving to someone similarly new to a thing like getting the flow mostly working i would say would take on the order of a week and getting it optimized to the point that astra is at with with that design would maybe be one or two additional weeks depending on the person so compressing that two to three weeks down to two days and change it's still mind-boggling like it shouldn't be this good at this as i would naively think about its training set but obviously with open ai's own chip development you know in-house they've i'm glad that those capabilities are getting added to the the models they're releasing to
Speaker 2the public and not just being kept inside we see jalapeno terrible name i think personally but you know their own chip development anthropic of developing their own chips deep secret supposedly developing their own chips we see the commoditization of the chip player with everyone
Speaker 1building their own chips how should we think about that as a consumer of all these things if i take my positron hat shirt off i would say that that's a great thing for the industry having um fundamentally that's going to bring costs down and capabilities up and and bring it to more people i think it's such an interesting world where when i got started in the semiconductor space you know 13 years ago you know silicon was a dirty word in silicon valley and now you have all the biggest companies in the world being you know somehow connected to the semiconductor industry and most interesting exciting applications and the company's building them you know vertically integrating down to the silicon layer the interesting thing with all the the ones that you mentioned and and you know the broader set is companies have the same macro goals the implementation details are all unique though and that's just as an engineer and technologist is exciting to me that there are a lot of different things that can be done in this space and so i think that's a great way for us to be able to kind of take a look at what's going on in the industry and what we're doing and what's going on in the industry so i think that's a really great way for us to be able to be able to be able to kind of take a look at what's going on in this space and i think that's a really great way for us to be able to kind of take a look at what's going on in this space and i think that's a really great way for us to be able to kind of take a look at what's going on in this space and i think that's a really great way for us to be able to kind of take a look at what's going on in this space and was where opening i unveiled jalapeno last month i'm happy that the industry is still pretty open and willing to share not as much details as people would have shared you know five six years ago but still a good amount of in the open discussion of of things and so i think how that applies to positron is you know we have our particular architectural views and way that we've decided to do things and that will evolve in the future as will everyone else's but you know there's still plenty of space to make bets and go in different directions and the great thing about the market is that the market gets to to decide what is is valuable and those that trade value will receive a reward for that we
Speaker 2spoke about context window length earlier and the expansion of it how much does that expand is that infinite expansion capability of context window length and what does that mean we can do that we
Speaker 1can't do today i'm just fascinated i think with traditional linear or quadratic attention there is going to be limits of scale what it came to what the hardware could provide now one of the things that we're trying to do with positron is we're trying to massively increase the memory capacity per device you know with our our upcoming generation we're going to have you know eight times more memory capacity than the highest memory skew from nvidia and nvidia is actually decreasing the amount of memory per device uh you know based on the market memory conditions but i frankly think that context length going from like a million tokens that it is today going to 10 or even great much more than that is really really hard with that quadratic expansion of memory cost the algorithm advancements that have happened over the past year have been very, very interesting in terms of being able to further reduce the amount of storage and compute necessary for that context with linear and sparse attention mechanisms. And those have really, really been innovated by the Chinese model labs. And this is a great example of when you have constraints of, you know, we had export controls on the chips with the highest memory capacity and flops. And so they innovated on not needing that. And so, you know, DeepSeq beginning of 2025 with DeepSeq v3 had, you know, made a lot of waves because they were able to get massive decrease in KV cache size with multi-head latent attention. So you were actually spending more flops to be able to have a smaller KV cache. And that's advanced a lot over the past year and a half. And probably the most interesting or my personal favorite right now is gated DeltaNet and its derivative versions. So, you know, you can have like a 75% decrease in the total time, you know, you're spending on, on the attention portion, you know, with, with this mechanism.
Speaker 2So how important then is new hardware if DeepSeq without it just on architecture innovation alone can cut costs by 80%.
Speaker 1Yeah, I would say that like, there's no such thing as free launch. So when they have that MLA compression, it does come at cost of model capabilities in some form. And like, there's a reason why the Chinese labs have really heavily embraced, you know, MLA while none of the U S labs have, I should say based on rumors, but I also have a very good information and belief that, you know, not, none of the major U S companies are, they, they definitely are not using MLA and not using some of its, you know, brethren. I think that, that, that will evolve and change in the future, but yeah, that basically the short version of that is it's not just a pure savings on that, on that side, but the reality is. So, you know, I think that's, that's a, that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. I think that's a really good point. seven, weeks old, only could do this about 70% of the time. GPT-6 Astra does it like over 95% of the time correctly. So there's a lot of room for improvement in these things. Speaking of room
Speaker 2for improvement, can I ask you, when I was doing the research for the show, I saw that the silicon data token price index dropped below $1 per million tokens this month. And five years ago, it was $60 per million. $60 to one. What does a million tokens cost in 2028, say two years from
Speaker 1now, do you think? I think the thing I care a lot more about than just that 60 to one is the fact that a $60 token five years ago, no one would pay a cent for today. Like that was a complete garbage token, relatively speaking, five years ago. And the level of quality for a token that you pay a dollar per million. For now is so much astronomically more valuable. And so that's because of token efficiency and what can be done. No, I'm saying just in model capabilities. If you, if you say, okay, so, so it's 2026. So the best model in the world in 2021 was GPT-3. It's kind of crazy at the rate models get released today that GPT-3 was the best in the world, basically from 2020, I think it was August, 2021, it released all the way up to, they, they didn't have a new release until 2020. So it's kind of crazy at the rate models get released today that GPT-3 was the chat GPT in November of 2022. So it was two and a half years between model releases. And really, GPT 3.5 was just doing reinforcement learning with human feedback on the same base model. So if you remember how bad GPT 3.5 was, and what was the value of that in terms of economic productivity value of GPT 3.5 versus GPT 6 today, or pick whatever comparison points you want, the value per token in terms of what it can improve a person's life, a company's business practices, et cetera, is orders of magnitude, I would say 100 or 1,000 fold. I think there's two points to your axis of going from $60 to $1. Yes, that's a decrease in cost, but that token today is, let's just say, I think conservatively, 100 times more valuable. So I would actually be saying that there needs to be some multiplier there as well, where the value per unit of intelligence is probably closer to 1,000 fold, not just the 60 fold you're talking about.
Speaker 2What does that mean then, if we extrapolate that out to 2028? What does that mean? Does the cost of a token then actually matter? Is that the primary unit that we should measure? Because everyone talks about cost of token. Is there actually a different metric that we should measure?
Speaker 1It is interesting that with the GPT 6 launch, Greg Brockman had said that he doesn't think that they're going to be pricing things in tokens much longer, and that they want to be moving to, you know, cost per useful result, like, to that. And I don't think that's where it'll end up, because that's really difficult to price and, you know, qualia, etc. But I think that the price per token is really great, because you can easily calculate the cost to generate a token. So determining a margin on that and pricing it in bulk volume to generic customers is really easy. And I think that's going to stick around in large form, because that is so easy. We'll see for the tokens, how they potentially evolve their business models in terms of if you have a GPT 7 or 8 that is superhuman and can fully function as an employee in an amazing capacity. And OpenAI calculates through whatever method that's running at full tilt, etc. It's only going to cost them, you know, however many hundreds of thousands of dollars to produce tokens continuously with that. They may decide that it's actually easier and they'll be able to get more adoption if they just charged a random number to have full unlimited usage of that virtual agent worker. So that may be how things
Speaker 2evolve. Can I ask you, I'm always very careful of being like the young, naive one. I'm not that young anymore, but like being the naive one who's not seen cycles. But Gavin Baker says it well when he says, I can't speak to a company that don't have numbers that are parabolically up and to the right and just everything is better than it's ever been. What would be the first signs of a crack in the chasm, right? A shift? From frontier models to open weight models, anthropic and OpenAI are not continuing in the same level, not quite growth rate, because it's impossible to stay sharp, but level of growth, missing numbers next year, and then the bubble getting burst a little bit, whether the two core leaders are having some form of strife.
Speaker 1I agree that that's a possibility. The reason I don't think it's likely is I think that the development of open source models and things happening locally, etc, will actually drive greater token volumes for the big guys. Sorry, how does that work? I thought they were competitive. Yeah, no, I think that the smarter and more capable that Siri is on my phone is going to result in it doing a whole bunch of background tasks and things that remove me from having to be the one that instigates having requests and data be processed by even smarter models. I really do think that in a lot of AI applications right now, the bottleneck is actually a human making some sort of decision. And different tasks have different levels of autonomy that will result in things getting sent to be processed by a model, by open AI or Anthropic. But I think the next really big order of magnitude, couple orders of magnitude increase in token volumes is going to come when us humans trust a local LLM that has access to all of our data all the time to have it decide to do things on its own that it is not smart enough to do. Right now, I trust Astra, a lot more than myself on a whole lot of different things. But I still prompt it to do things and maybe it will go run autonomously for 12 hours or three days. I think the next big leap is when I trust a model running on my laptop or on my phone to prompt the smarter models to do even more wider set of tasks.
Speaker 2When we look at like the data economy, the powers, the larger models, which you believe in, we see Mercore, we see Surge hitting 3 billion in revenue. Yeah. How big do these companies become? Because Anthropic and open AI are four to $5 trillion businesses, say, feasible. It's wholly feasible, isn't it? That Mercore and Surge are $200 billion businesses, which serve both Frontier Labs and some of the world's biggest enterprises building their own models.
Speaker 1I think my only skepticism there is on there being vertical integration by the Frontier Labs. I think the reason that hasn't happened is because Anthropic and open AI, they have better uses of their capital and mental power, et cetera, than doing all of the scale AI, Mercore, et cetera, work. But I think that won't always exist. And let's say that GPT-7 or 8 could be an effective replacement for Sam Altman in terms of being able to manage a large business, why they wouldn't just have agents be taking over those tasks.
Speaker 2Before we leave, I'd like to finish on a tone of optimism. What are you most excited about today that you think the world does not spend enough time on that we should spend time on?
Speaker 1I'm very sympathetic to the problems that I think the smarter set of the AI alignment, AI safety community are when it comes to thinking about how do you align incentives. A lot of people just talk about AI alignment being a problem. I think there's a key part of that is human alignment. It's like, how do we, society, human race, align ourselves to have a good outcome that I think will be empowered by artificial intelligence? And so we discussed a bunch of the different problems that we're facing geopolitically and socially and how these different things are handled. And I think a lot of smart people are doing good work on the AI alignment problem and thinking through how we solve that. But they may be gated in what can be done there if we don't get better human alignment on, be it regulatory frameworks and energy production, where we're going to put the data centers, et cetera, et cetera. And I think framing it as this being a similar sort of technical problem that smart people can work and reason through will hopefully get more people thinking about it that way. And I think a core element that gets discounted by, I think a lot of people in that sphere, there are a lot of economic factors. And I think it's the economic factors that actually will drive real decision-making and actions. And if we don't look at it from a rational, self-interested actors and all these different things, you're just not going to make a difference.
Speaker 2Thomas, this has been the most varied discussion ever from education on unbelievable infrastructure evolution to Dario and Sam, you are a star. Thank you so much for joining me today. Thank you so much, Harry. But before we leave you today, the best model for your application might not exist yet. The most ambitious AI teams are training open models to beat the frontier in their domain. Now, on November the 3rd in San Francisco, Forge by Fireworks brings those teams together. People, like Jensen Huang, CEO of Nvidia, Michelle Kataster, president and head of AI at Replit, Lin Quao, CEO of Fireworks and more, to hear how AI leaders are taking control of their differentiation, margins and roadmap by owning their intelligence. Learn how they're building it with inference, training and intelligent routing. Forge is free to attend, but space is limited. Applied today at fireworks.ai/forge. While Fireworks AI powers product intelligence, Asana keeps the work moving. Most companies have tried AI, most aren't seeing results. Not because AI doesn't work, it's because AI hasn't reached the workflows yet. That's the gap Asana is built to close asana is the operating system for human agent teams your easy button for ai productivity across every team ready to go ai teammates pre-built for marketing ops and it no prompt engineering no setup they show up where the work is happening already onboarded in your workflows ready to deliver with asana your whole company can work on the same plan towards the same goal whether you're a team of 10 or a team of 10 000 asana where humans and agents workflow together try it at asana.com that's a s a n a.com while asana organizes the work superhuman speeds up the day can i be honest with you for a second here's what my week actually looks like back-to-back meetings all day and the real pressure builds up in the gaps between them i'll come out of six founder meetings with a whole stack of follow-ups waiting and my inbox it's hundreds of messages and somewhere in there is the one thing that really matters which i always tell my mother is her i've had ai tools for this but they always lived in another tab i'd have to stop go find it paste in the context write the perfect prompt it was just enough friction that it never really stuck super human go from the makers of grammarly totally different it's right there where you need it when you need it in the doc when i'm reading the email i'm writing when a thread gets dense i highlighted and asked go to pull out what matters without losing my place and after a run of back-to-backs i asked what was actually decided and go turns it into an email and i'm like oh my god i'm going to have to go back to it and it turns it into clear action items or drafts the reply i need to send it's already up to speed and i'm always the one in control try superhuman go from the makers of grammarly and find out more at superhuman.com
PodcastTranscript AI