Solana's Biggest Upgrade: Alpenglow | Roger Wattenhofer, Kobi and Quentin Kniep from Anza

[MUSIC] Welcome to the third episode. I'm your host, Shobbik Deb. AI is making intelligence digital. Crypto is making institutions digital and digital intelligence requires digital institutions. The mission of this show is simple. Talk to the most fascinating entrepreneurs and intellectuals shaping this future and extract lessons that you can actually apply. Today's episode is about one of the most ambitious protocol upgrades in crypto right now, Alpen Glow, the redesign of Solana's consensus and data propagation architecture itself. To explore that, I'm joined by Roger Wettrenhofer, Koby Silvensky, and Quentin Neap from ANZA, the team leading the design and implementation of Alpen Glow. Roger is also a professor at ETS Zurich and Koby and Quentin previously did their PhDs in Roger's group. Roger, Koby Quentin, welcome to the show. Hey, where's we? Thanks for having us. Today we'll cover three things. First, what was broken or limited about the tower BFT and so proof of history. Second, we'll go over how Alpen Glow actually works. And the third, what these changes in practice for validators, for MAB, for developers, when finally they call up says to hundreds of milliseconds in the Solana ecosystem. So I want to start with the story. So you guys studied Solana's consensus from the outside. We do wrote a paper about that, that how Solana could be halted, the previous consensus architecture. So how do you go from writing a paper on how to cause Solana's consensus to actually building, but maybe the largest redesign of Solana's core protocol since the launch. I can take this. I mean, we had this student project based in the university. Still, we didn't only look at Solana, we also looked at other blockchains. And we just tried to find maybe things that could be improved or holes. And yeah, we found something we thought was a bit wrong about Solana back then. The then, you know, the founder, Solana found herself contacted, I think, Koby, right? For some reason. So yeah, maybe Koby. He contacted Koby and then we kind of got the block with each other. They sort of knew this problem already that we were describing in the paper. It's not, this is not news to them. So they had already some, let's say, bug fixes about this problem. But still the founder was very open to talk with us and we discussed the protocol a bit more. And yeah, eventually we got some homework from them to show that we are worthy. And eventually they hired us. That's sort of the story, I guess, but Koby might know more. I think it was a pretty interesting meeting of very different worlds. Because we come from mostly right, we did PhDs and we write papers. And we talked to academics and they think about theoretical problems, right? And Solana was known and I mean, it's still known and very much into engineering and problems that are painfully real. All right. So at the point I want to make is that they thought about like some properties they want to exhibit in practice. I want to mention, for example, like really pipeline distribution and execution of box. Okay, that's I think something that Solana pioneered. That's really nice. And I haven't seen this kind of thing in academia anyhow. My point is they have their practical properties that and they did that. The sort of theoretical rigor behind it was neglected. And then we came in and we said, oh, look, there's this, there's this thing in there that's not so nice to read with me. Right. And then I think it was very nice that we got to talk to people who. Like as I said, they know and now we do know and we think about problems that they're in practice and very important in practice. But then we can also contribute to that it's now. Theoretically sound I see. And is that the origin story of the whole Anza Labs also like you knew like when you guys moved over that's like you bootstrap the Anza Labs itself. I mean, Anza existed already before. So this is this is just not this is. You know, about 50 I would say people this is kind of a known the people who are behind the agave protocol which is the main protocol for Solana. And basically happened before and was kind of independent of joining right we just joined on the point because Anza was already going to be the place where agave will be developed. But they did call us the research lab of Anza then right. So that's sort of like that was new somehow. So even that we are the only ones with PhDs, but there's not too many I would say in. Yeah, in Anza lower PhD density, but maybe other labs. Okay, so maybe it's one trial to talk about. Given this origin story that what was broken in their like previous protocol in the proof of history and the tower BFT what exactly did you guys found and it seems like they already knew about it. But what was exactly the problem? Well, I mean, so the current protocol basically produces blocks are there slots it produces blocks. Some leader has opportunity to produce a block for a slot and the thing they neglected early on in some sense is that the leader could just. Acquivocate to have two two blocks for the same slot essentially in some way. This is also to do I would say with with how protocols are often specified right so when you go to actual companies then sometimes they the code is this specification somehow. But we kind of looked at specifications we could find and there was this cry still in the old way. Yeah, but that's how it's a root of the problem. I think one reason they might have neglected the somehow is probably also turbine right because. Turbine at least makes it harder or seems to make it harder to to equivocate for the leader. And you might be yeah, you might think that it's enough and you can rely on that but you you still need to handle equivocations even with with turbine. Isn't the turbine like modular enough like it's like a data dissemination protocol right just different from the major consensus part that is going on the tower BFT and the POH. Is it not easy to. That analyze them separately through what can't in says is that they didn't worry about it so much because turbine would solve the problem. You know like you you could argue if you have a good data dissemination protocol which has full tolerance built in right then you. Then you send around stuff and if you really equivocate on the same slot then you. And people would notice that somehow right it's not it's very hard to do that actually given that this is the motivation we should. So it's the girls to talk about how I'm going to work this with the two things that's a lot of cares. Much about right I want to have a many transactions per second and fast confirmation. And they want to have a good security model so we added something new here which we call this 20 plus 20 security model which is 20 Byzantine. 20% Byzantine stay can 20% crash phase on top of that which is somewhat different from the typical 33% that most others have. So in some sense a bit worse for Byzantine failures but a bit better for crash failures. So to just summarize on this the three three things that you said these are the three parameters throughput. They want to increase the throughput decrease the latency does the second parameter. And the third one is about this security resilience in which you have added now a crash fault. Nature to that it's not just Byzantine but it has Byzantine plus crash fault. Now I would say the first two where you know this is like just the motto of Solano right they care a lot about this increased bandwidth and use latency that's. That's what Solano is about I would say. And the third we we added right but I think it's kind of like we didn't add this because we wanted this so much we thought about it and like the idea to have crash failures on top of it because. Crash failures seem to be a lot more common in the real world than actually Byzantine failures at least. And we look at how blockchains are used these days. But. Crash fault is something that is not generally cared in the in the academic land the am I getting that one correctly and in the real world it is much more prevalent. I think it is you know I'm talking as an academic year but. But we see crash faults all the time but you know like when was the last time you actually saw Byzantine attacks right I mean it's good to be tolerant towards Byzantine text don't get me wrong but. But you don't have to be you know I mean one main problem one main difference is probably also that for Byzantine attacks you can in most protocols you can attribute them to to the attacker right. And then you can slash them you can punish them that that greatly reduces the incentive to actually try to do such an attack if you already know that the outcome then you are. time from the network. or slash children and lose all your stake, then why would you even try it? I see. But does the accountable safety part, I guess, if you are malicious, you know? But I think it's a good point and a big picture, sort of interpretation that it's usually in academia to think about faults. Like all of them, right, in one category, be the crush or Byzantine. But as you say, the crush faults are much more common. And then this is one of the things where we like, okay, we should optimize for what's the problem in reality more often. Yeah. So I mean, this is sort of like these are the four main ingredients, maybe that Solana had before out and go. And we kind of remove all of them, except turbine, which we liked, we kind of replace it with something a little bit simpler. But this is still related, I would say. And then we added, you know, so we looked at the recent work, basically, we were inspired by some recent work on consensors. Here are three of the papers. And we added a voting component, which we call voter to this rotor. Okay, so these are the two new ingredients. And in some sense, it's a big change if you want, right? So we kind of changed, we removed three of the four parts, change one part, and then other part. So the three of the first, so if I'm for our listeners who might be listening, the four parts are proof of history, gossip, turbine tower. You remove proof of history, gossip, and tower. You retained a simplified version of turbine, which we call, which you guys are calling a rotor. And added a new part called voter. And the voter is, is the consensors part, is that correct? Yes, the rotor is to date is dissemination part and the voter is to consensors voting. So rotor the data dissemination part, just how this works. So it's a simple leader-based protocol. So we have blocks, which contain transactions. So the blocks are just, you know, sequentially sent out by the leader. The leader changes over time, but that's basically how it looks like. And we do consensors each block. But then each block is pipeline into what we call slices. So these are like, you know, maybe just a millisecond of data or something for the data dissemination. And then the slices are split up in shreds, which are so called coding shreds and we have. So for each slice of data, for some amount of data, we actually had doubled the number of data in the shreds. With the idea that if you get half of the shreds, then you can reconstruct the slice. And if you can reconstruct all the slices, then you can reconstruct the block and then you can do consensors on them. The way to think of slicing is it like, suppose I zoom that block as 10 transactions. And slice one can have the first three transactions. Slice two can have the transaction four five six. And the slice three can have the like the rest of the seven eight nine dead. Is that the right way to think of the slices except the single digit now to break my heart a little bit. Yeah. Is there a slicing algorithm or something like that special? But it's just some set parameters out there. Well, the leader just collects transactions basically and then sends them out together. That's what the slice is. I see. I see. So the leader has all the power to decide and like how to slice those things. Yeah. Like it's up to the leader, like it's up to the leader scheduling algorithm to actually build the slices. And then in the protocol, it's just an upper bound on the basically on the number of bytes that you can have personalized. That's then how we distribute it. Right. We have some fixed number of shreds. I think right now in the main method 64 shreds and the state or somehow has to fit inside these threads. The idea is that each each red basically is a packet in the internet. So it's based the MtU limit. Yeah. And this gives you the limit on the slice. Each shred is supposed to be just one UDP packet. And so that there's no fragmentation on the on the network layout. I wanted to mention that it's usually the leader decides this kind of like slicing or ordering or whatever. But I thought it's worth mentioning that it's an active area of research for us. And we want to change the game in this respect. So yeah. So the leader sends basically these shreds. Now this is a single slice here on the screen. The leader sends the shreds to the what we call relays. And they basically just send then their shreds to everybody. Okay. So that's what they do. Let's say there's a thousand validators in the system, something like that. They were just every leader were just same test rate to a thousand other nodes in the system. This can be done naively. This is very fast because it's all just UDP packets. But it could also be done with some multi-cast service which exists in the salon of a world like something called double zero. So generally in blockchains, almost all blockchains, they have the leaders trying to disseminate the blocks by themselves. Instead here, it's disseminating to a something an intermediary call race, an intermediary set of nodes called relays. What exactly the problem it's trying to solve by doing this? Is it like some kind of bottom-led throughput? I just want to say this is such a good question. And it's been neglected for a long time. I mean, the way you said it, you said that in some protocols the leader disseminates the block to everyone. And there are some protocols where this happens. I mean, what's more usual right is that you gossip the block. But either way, there's a problem. Like, if the leader sends the block to everyone, I mean, even if there's just a few nodes, like even if there's 20 nodes, not to speak of thousands of nodes, if the leader is supposed to send the data to every single one of them, like literally, right, then you immediately get one end of the bandwidth of this leader, like, per node. That's terrible. It takes a lot of time. While in gossip, okay, you might think that it helps. Maybe, I mean, it kind of helps, but not too much in the sense that, okay, now you have many steps. The leader sends the block to some nodes, then some nodes send the block to some other nodes. And again, you have many steps. The problem with both of these scenarios is that most of the time, most of the nodes do nothing. So if you want to say it simply, what's really different in here is that most nodes, most of the time, continue their bandwidth. It's a huge difference in practice in the sense, I mean, with respect to how fast you're able to disseminate the data. Yeah, and it's very full quadrant as well, right, because you don't need all the shreds to put the slides together, right? Only half the shreds are necessary, let's say, if you have more provisionary vectors, so it's full tolerant and fast. And it's even more fast in some sense, if you think about it, so the leader sits somewhere in the world, he's relased it somewhere in the world, and then if you only need half of them, then maybe the lowest paths you don't even need, right? I just need to pass the fast path. Yeah, because if you just look at the protocol, you think it's two network hops, right? But in practice, quite a bit faster than two network hops, because you get the fast thing that you guys are saying, this is something like forward error correction, where you try to find the, like this is what content delivery networks do, right? With forward error correction. So is that is that the similar principle here? Yeah, yeah, same. Okay, okay, my second question is, for, suppose I'm the third relay here, I'm running the relay, how do I know which, which, like do I know which data shreds I should get from which slice? Like is there some kind of global understanding for each relay, which one they're getting, which shreds from which slice? They wouldn't need to know it, the relay is actually very dumb in some and said they're really just forward whatever they get. I guess, I mean, at least the leaders, of course, know who they are when they have to do something, because they have to initiate something. The relay chosen randomly, they don't even know that, right? The leader could even choose their, their best friends to do that for them or something if you want it. So it's not necessary. You guys are mentioning that it's much more fault tolerant. So how does the system ensure that this, that this is not just the leader choosing their own friends, but it's, it's robust, robustified to a certain extent. Sometimes I would say that the leader could choose is the most robust thing because the leader really wants the block to go to everybody. So to choose your friends, hopefully helps here, but I mean that this is not the way we specify the protocol. And this is just something, need to say that there's a lot of freedom here, what to do. And this double zero multicast network, which is already something like 40% of the state, Consolana is kind of connected to that. So there you can, for 40% of the state, you can actually send them directly from the leader via multicast if you want it. You can do this on top of this thing here, just to know to have even more ways to get the data to the, to the validators very fast. I see. So, so this double zero connection that you're mentioning multicast, it's a separate one from the relay, it's relay system itself. Yes, yes. And since it's all a ratio code that you could use both for the same time, right? You could say as a, as a user, no matter where I get my shreds from, from double zero or from this rotor, I can put them together into a slide. That's fascinating. I don't think any other blockchain is doing this kind of efficient like dissemination right? Is the Solana is the only one I'll pay with the Alpen Goa? Possibly but I think this is picking up speed right now so I've heard others which are also thinking the same. I want to brag about this I mean I shouldn't brag about this because I wasn't there to like to it but Solana really pioneered this technique and in academia it gets more and more recognition these days. This you know erasure covered dissemination in blockchain protocols. Yeah already turbine did this you know but the only difference is that turbine has multiple layers which is something we felt is not necessary but makes things more complicated and also it's lower right brings it closer to gossip somehow which cobi's at the fore makes things just slow right because you have to send the packet multiple times until it reaches destination. Are release here incentivized or is the assumption that they're altruistic? Well they should be incentivized in a proper protocol whatever work you have to do for the protocol you maybe should get something out of it so they should be proper incentivized. That's the plan for the future problem guessing. We have something already in place but it's not as good as you want it to be maybe it's not as complete as you want to be. I just want to say that this is a good question I mean you're saying oh they should be a reason why the release forward the data. Yeah I just want to say that it's not specific to Alpenglow or to Solana this problem I mean it's very I mean okay you could ask why would let's pick some other I don't know Ethereum why would Ethereum not gossip distance such and such data. In every blockchain protocol some parts are not going to be incentivized and it's a problem and we should think about it and it's a good question I just wanted to say it's not specific to us anyway really. It's probably a general peer-to-peer question it's like how do you like why do you propagate your data to someone and why do you behave as a relay to others right. Good so this is the data destination part the voting part so yes so here we can have the blocks and you can see here in Solana it's basically currently each leader does four blocks and then it's the next leader's time. I then they use timeouts so all the validators basically are referees just to check whether the block is in time so this is the idea from simplex so you have some time this is really local time right the nodes just have their system time or something they know when the last block from the last leader for instance was received and then they start a timer and they say at that point I want to receive the last block from the next leader for instance and then they can just vote yes or no whether the block was in time or not in time they don't do this just for the last block they do it for all four blocks of the next leader and this is important because if the first block is already late then you might get rid of this leader very quickly if the if this leader does not work well so then you basically vote yes or no on these blocks let's say we vote yes in my example here so this is also something we just basically multicast out again right so it's just we just send our vote to everybody this is we don't have like a million you know validators you only have maybe a thousand or something in Solana so it's a bit lower they voted to send this to everybody they vote so there's a few case of what happens and now I take the I received the votes everybody sends their vote to everybody so everybody receives the votes from everybody and if you get 80% or more of the stake voting yes then this block is immediately final okay then you can immediately build a certificate which is a BLS type certificate and the can immediately say okay here's a proof that this block was final so within one round of voting if you want and then you also send these certificates to everybody so that everybody agrees or see that as well then the other extreme is that you get a lot of no votes because the block was too late for instance let's say you have 60% or more of the stake voting no then you can immediately build this keep certificate which says we skip this slot we skip this block basically also that you can send to everybody and then the only interesting case is between those two cases essentially and you have like a mixed thing between 60% no and 80% yes there's many possibilities there and in the mixed case we have to do one more round of voting so basically it's not yet final we cannot make a certificate right away there's one more round vote and to restate what you are saying there are two types of consensus protocol one is where in every slot or in every view the leader rotates the leader changes but in in the Alpenglow it's more like each leader has a fixed number of slots it's not like every slot it is changing in the example that you are showing it's like four slots the same leader is there and then next four slots another some other leader is there it's something like that is that correct to say that yes this is something we inherited from the previous protocol so to speak right like whatever we could keep you of course wanted to keep and it felt like a good trade off to do it for a few slots however what we have what is new here is if a leader is not compliant then we can switch to the next leader much quicker right that's in something new let's take the four slots example I want leader and in the first first slot if the in the block does not come if the block is not proposed by the leader then the whole system says we should just move away from this because this leader has skipped the first slot and we don't trust it anymore let's just move to the next leader as soon as possible yes that's the there's the philosophy here right that's very much so skip means skip for the rest of this leader essentially we go directly to the next leader there are like bunch of protocols where like previous consensus protocol where two or three two or three rounds take place where when you one block is like it's being is being uh is being voted upon but after that finishes then the next block voting starts in alpango is it like it does not wait for the whole uh consensus for each given block to finish but immediately after the first slot a block is proposed in the second slot another block gets proposed is like very pipeline is there an optimistic block construction is happening or something like that is happening here if you have been the leader before in some sense then you can just continue piping out stuff you will trust yourself that your block gets in right so you can be optimistic so the only interesting case in some sense if you have a change between leaders right and yes then you will do something very start already producing things as a next leading that the previous block gets accepted at some point you have to say who your parent is right again if you your your own parent if you read the leader before then you can trust yourself that this works for you but if you're the next guy then you would say you know uh then you wait basically we put that in the block but you can block take 400 milliseconds so there's a bit of time to put this parent relation in there so you were asking if we vote on blocks sort of concurrently right and I just wanted to say yeah we do like you don't wait for a block to be finalized only to be allowed to vote for its child there is an like some other block so we do have you know your vote for a block then you might not see any decision we respect to that block yet and you might already vote for the next block on the next block my understanding of the optimistic was in the corner case which Roger actually touches when the leader gets changed suppose suppose uh suppose leader one four slots done and then the leader two comes for the next four slots so it has to make some guess that oh it has to tell what is the parent right something like that and yeah there we have a small optimization which does that which actually where you actually start building a block on one parent if you like if you see a block in slot four you you start building a child on that block and then if you later see that consensus actually decided on another block then you would basically switch the parent and basically restart your block kind of okay so and the switching protocol is a is a different part like how do you switch I would say it's like a small extension you could okay you could do I can go without the small extension and we just put the small extension in there but it to say it's a very good question right and it relates to whether it could or could not be possible for example for a leader to kind of treat the next leader right maybe something funny happens and then the next leader in the block but then it turns out the ancestor of his blocks was actually decided against it was rejected so it is a problem in some protocols right if they don't address it right it could you could have this thing where one leader might potentially treat the next one but yeah I mean we we do take care of that and there's this I mean what Quentin mentioned right this optimistic that okay it's optimistic for a moment but then okay the leader the next leader after it does the thing where it might or might not switch the parent ready it has a guarantee that its blocks are going to be accepted I mean if it's correct and everything that's true this was the the the block proposal parts so I want to go a little bit deeper on this finalized condition aspect for each block. The first one that you were mentioning that it takes 80% if 80% people's water say, yeah, we agree on this. Immediately it gets finalized. So this is some kind of fast finalization. Right. And suppose 80% don't agree-- so is this like parallel-- some kind of parallel fast parts, slow parts protocol. Is that something like that? Yes. Very much. So what actually happens is-- so you're sitting there as a validator and you collect the votes of the others. As soon as you see 60% yes votes, then you already send out the second round message, basically, which is to-- which will be the finalization for that-- the finalization message. But then you keep collecting votes for the first round, so to speak. So now what could be-- it could either be that you reach 80% for the first round or you reach 60% for the second round as well, because all this also have to send this out. And whatever is faster for you is what you get the finalization. And it could be that for you, maybe 80% is faster, but for me, 2 times 60% is faster. This is quite cute, actually. And we actually do see that in practice. There are notes on the Solana main network, one is faster and the notes where the other is usually faster. But the protocol is safe enough that-- suppose one guy gets the 80% faster and the other guy gets the 260% faster, they will still agree finalize on the same block. That's always after the same. If they didn't agree, the protocol would come with circuits. [LAUGHTER] But I really want to hammer this in because I think it's pretty counter-intuitive, right? That in the same execution, for the same slot, for one node, one is going to be faster. And for some other node, the other is going to be faster, which is-- yeah, I think in theory, it's often-- I think it's very important in practice and type this. They often forget about it. They really don't even look at it. Yeah. Theory, it would be difficult to even analyze this because there you usually have these rounds, right? And you say, how many rounds does it take? And then people would say, well, obviously, the one-round protocol is faster. But here, the two-round protocol can be faster in practice, and actually may be often is. One thing you mentioned, there's a concept of timeout for leader rotation, all this thing. Is the-- and you're mentioning that this timeout is based on each node having-- using its own local clock for timing out. There's no concept. There's no need for a global clock. And Solana had this global clock called proof of history. That proof of history is no longer needed for this. It says, locally, I'll just time it out. And other nodes will just use their local clock to time out. Is that the notion? Yeah, it's only local clocks in this part. You know, local clocks do drift a little bit in what mode, but it does not really matter, right? And the grand scheme of things, because what matters is that we have 60% good nodes. Otherwise, if we have less than that, it will be a problem. This was an early latency histogram, which kind of summarizes the whole thing. This was for a leader in New York. And basically, the green curve would show for 100% of the stake. So it goes up to 100. How far they are from New York? This is just network latency, let's say. Maybe this is 200 milliseconds. So some people are very far from New York, some of the validators. You can see here some geography, right? Some European nodes, some American nodes, how far they are from New York. And on top of this, we show how much slower road there is, how much overhead it adds. And it's not much, you see, for some nodes, it's really just basically the network latency plus a little bit. For some, it's more, but that's roughly it. And then you can see how long it takes until you have 60% of the votes in the good case. It only shows the good case. And then it shows how long does it take until finality. Whereas finality here could be 80% or it could be 2 times 60%. Or it could also be that you just receive certificate from another mode that it has finalized, right? It's also possible that this could be the fastest thing. So whatever is the fastest thing gives you finality and this kind of shows that the overhead of the protocol is roughly maybe two times the network latency or so. This was the whole architecture. Now let's talk about the incentives. Because I'm sure that the economics are changing with what validators have to do here. What relays have to do. One of my question is, so previously votes used to be put into the on-chain, right? People were voting on-chain the votes. Now the votes are not on-chain. They're being gossip like all to all propagation is happening across the peer to peer. It's off-chain happening. So-- And that means your-- the fees from the voter transaction is not there anymore. Your leisure is not growing that fast. This definitely changes how much work validators have to do for storing data, for what kind of verification they have to do because the valid voter transaction is not there. Can you talk about how does it change the validatory economics in general? Does it make the validator running a validator software cheaper? No, these are good questions. And somehow these were the most-- you know, when we present the open-gloves, of course, the validator is only cared about the parts you just asked. How does it change our economics? So before they had to send votes to the chain, and that actually cost them something to send votes to the chain. This was like a transaction that the paid vote. And they would get rewards back again in some sense. But if you were a very little validator, then maybe it would not even be worth to send the votes to the chain because it would cost you more to do that than to get the rewards. So now the votes are just sent directly. There's no more cost of the block chain in some sense. So we have a lot of votes for votes. And this relieves them of this cost, which would have cost. However, we didn't want to change the economic game too quickly immediately. We wanted to kind of keep it a little bit similar. So what we now have is we introduced this VAT, it's called-- I'm sort of a tax essentially-- if you have to pay as a validator to be part of this. So every epoch, every two days or so, you have to pay a small amount of money to be part of this. This is less for the vote used or not. But it's in the same ballpark so that we kind of could argue that the economic game didn't change very much. So this is for smoothing out this transition, this tax. And eventually, I'm guessing this tax will be removed. Co-pere removed or more, you know, like that we have real economics in some sense. So let's put it this way. So this is a bit my view. But the question is a bit how many validators you want to have to make the system secure, right? So if you have, let's say, a million validators, then, of course, there's more protocol overhead, right? Everybody has to get the data. Everybody has to get whatever, right? You have to build around this. And let's say if you would remove this tax completely, then maybe we would have a million validators, right? And that would be problematic for us in some sense, right? Not that it wouldn't work anymore fundamentally, but it would be slower or something. We don't want that. So it could be that we just find the right economic incentives eventually. But, you know, like, as is every blockchain is governed and so do you have to ask the people who are the validators whether they agree with this update? And we felt if you make it somewhat zero sum game for them, or maybe a big different incentive, but not much, they would still agree to the protocol change. And then we can then tackle this later. OK, this is the real economics. Suppose the next epoch is coming. So in the previous epoch, you have to pay the tax for participation in the next epoch. How do you think about this? It's exactly what you say. You have to put up the money first, so to speak, to be part of this. It basically becomes part of the epoch change mechanism, right? When we determine the set of validators for the new epoch, then we take this into consideration, right? We deduct the fee that is required. And if you cannot pay the fee, then you are not in the next validator set. And otherwise, we take all the validators who pay the fee and make that the new validator, et cetera. The other question is, how does the MEB dynamics get changed? Given that now the finality is in the order of hundreds of milliseconds, so you have to be even faster. I mean, that's probably like to even extract MEB. Does that-- does that to centralization in that case? Because only bigger ones can do it, and they only can participate for doing this. But on the other side, there was a goal for reducing the cost for a running validators so that it gets more decentralized. Seems like there is a push and pull happening. - Well, I mean, decentralization is super important, right? And it can mean various things. And so I would say, it's not clear to me why something is more decentralized. It's very dig validators and tiny validators together. Okay, so but let's say geographic decentralization, of course, does matter, right? I don't want all of them in the same location for security reasons and other things as well. So we tried to build a protocol around the idea that you would not have an advantage. We tried to have to build the protocol so it says no advantage by sitting at the right place, but you can only do so much. Let's put it this way. - I just want to say I sometimes here, or we sometimes hear this like a concern or this association that something about MEV gets worse, but the way I understand it, it only gets better. And for a few reasons. So one reason, okay, you mentioned latency. Actually, Alpenglow equalizes the latency between like validators that are in some cluster. Let's say in Europe, that's where most of the stake is these days. So Alpenglow evens out the latency differences like going from today to Alpenglow for the remote validators. So the latency should improve more. Then it improves for the ones that are already somewhere in the center of the market. Okay, in this sense, yeah, I think it's only better for remote validators. That's one thing. Another thing, so Tower, there were some new ones like MEV's behavior is that you could employ to take advantage of some other evaluator, right? And the better connected you are, like you threw in the, again, in the center of the universe, maybe you could play these tricks better. Now Alpenglow addresses some of these problems. So again, I mean, the only ways in which I see the landscape changing for NG, I think it's for the better. - Here better means more efficient extraction of the SAR plus that might be there. No more efficient extra, I'm not sure if that's what they meant. So, I mean, you mentioned centralization. Okay, so centralization is something, it's a fourth where if you have an optimistically response, it's protocol and there's a cluster that is well connected. And the latency for this guy is gonna be better. Then some modes in remote wherever in Antarctica. Okay. Then you're gonna argue the node in Antarctica has a disadvantage, right? So maybe it gets less money for this or that reason. Now, I'm saying, okay, it's always gonna be the case that if you have a node in Antarctica, it's a bit worse for you or like maybe you get less money. Now, I'm saying for Alpenglow, this difference is gonna be small. It's gonna be better for the guy in the dark type data. - The, the force of centralization, I mean, in the center of incentives, there are any V, like it's pretty much in every protocol. You would, you would have to get really as a theory to equalize it perfectly. But Alpenglow makes this difference smaller. It does not exhaust your ability to improve it. - Now, now that we have talked about the economic side, we want to talk about the integration with the other components of the Solana ecosystem. So you have, you have other things like double zero, multi-multiproposar architecture, all this going on. So my first question is, how does this interact with the, like multiple concurrent proposal? Does the integration is very straightforward? Or is it there are nuances to this integration? - It depends a bit who you ask and what kind of solution we are making. Right. We have a few proposals in the pipeline. Also, people start already programming things, testing things. I would say for all of them, Alpenglow is somewhat an necessity, okay? You would not want to do it with the previous protocol because it's just too complicated in some sense. The simplicity of Alpenglow helps to make this happen. And some integrate very nicely with Alpenglow. You have to do very little things and they can already code have this multiple concurrent proposals. But this is true. This is, I would say, the other big project that Solana for this calendar year. So Alpenglow will be released. I would say for sure in the first part of the calendar year and then shortly after probably we'll see a multiple concurrent proposals solution as well. - And this MCL solution is just, just plugging on the top, whenever the proposal is, leader is proposing the block, just before that you get it from the multiple proposal. Is that, is that, is that simple or is it? - Some of the proposals are that simple about there's, there's real four things, right? And we're still discussed which ones we're going to do. And no matter which, how we're going to do it, you know, it's the first come up with a very simple one. But I would say the very simplest ones we have are of that kind that we maybe just have multiple proposals, multiple leaders at the same time and then vote on, you know, what have you seen or something. It's quite compatible out there. - Out and go. - Next question. What, what needs to be changed at the RPC or the web socket level? Do they have to upgrade their infrastructure for Alpenglow or is it same as before? - Yeah, that's also a good question. Well, why do I have to take the hard question? - So in some sense they don't have to change anything. So if an RPC is willing to pay this VAT, you know, the fee, and they can just be part of the protocol that, you know, there's nothing they have to do. They can, they are first, Edison's in some sense get it as fast as anybody else. If they don't want to pay that, then then it will change for them, I would say. And they have to get the data from somebody else, which is also possible, right? They could get the data essentially second hand. From any set of validators, they get its first hand or something like this. You can offer it as a service. Also there we will have several paths, I would say for success, depending on how good, you know, your service should be and how much you're willing to have discussed you, but there will be very low cost versions of this as well, maybe zero. Many RTCs are maybe, you know, also somewhat affiliated with some validators and tell it will be very simple, that I just say, you know, whatever I get, I forward to you. - I saw somewhere that few of the Solana programs, they need some notion of global clock and POH is being removed. So what is the plan for those kind of Solana programs? Like do those app builders have to rewrite their app or is there something that Alpen Grove will come in built that will help them in navigating this? - Yeah. So we will have something like a clock still, so something like a real time clock, right? This is something which now is for free basically, because when you vote, you would also tell what the time is and then everybody sees what the time is. So to speak on chain, this will not have any more because the votes don't go on chain anymore. What we have is simple, you know, I would say, approximation of this somehow, the idea is very simple, the leader will just add their time to the current block, just add the current, you know, however they see the universal time currently. And basically they are not allowed to deviate too much from the last from the parent block time, okay? So they, you know, they should be honest, but let's say if the time is a bit behind, they cannot catch up too much or they cannot slow down too much, such that if most leaders are honest, then the time should always be around the right time. But if you're unlucky and you have let's say 10 liters in a row which all want to slow the time down, 10 malicious leaders so to speak, then this could also happen. Then it slows down a few seconds and then the, you know, good leaders coming in will speed it up a little bit again. But so it will not be a super exact time, but we believe for the applications that are out there, which usually needs the kind of correctness of about, you know, an hour or something like this, it should be good. What is the planning for this migration? Ethereum data similar kind of transition back in 2022, which are like a lot of detailed planning on, is this something on those lines? You guys will have also detailed planning on how to do this migration. - It's a hard problem, I would say, generally to make this happen, right? So this is, as you say, it's a huge update for Solana. It needs planning, but it's not that we have exact timeline now. I went on stage at some point and said, "Media this time, you would already have it." But that was maybe a bit too optimistic. I would say most of it is implemented. the client of Anza. So in the Agarve client, the implementation is roughly complete. I would say it's running already in some smaller test nets. But some of the corner cases are maybe not yet fully implemented. If things go really bad, right, you also have to make sure that we have those. But we have a plan how to do the migration. I know how to switch from the old protocol to the new protocol. And this has to be like this, right? So the protocols are very incompatible. So everybody has to switch at the same time essentially. This is a, this is what say is a difficult thing, right? So in distributed systems, this is one of the most difficult things if you have a thousand notes, which all have same switch at the same time safely, right? So that's sort of the, I would say, that's something we have to worry about, but we are not alone, right? There's a wee. Any tentative date on when this switch might happen, like this quarter Q2, Q3? No, it should, Q3 might be late, I would say, but it depends very much on the testing phase, right? So we will have to extend the testing on this, on some test net before we'd actually go to main net. And now this could be faster or slower, in my opinion. So I don't think once the testing is done, then it can go fast, right? But until then, I'm a bit hesitating to get you a date because it could be that we run into some issues that we have to solve then. We covered a lot today. So we unpack, I'll pan-glow as a system and what this means in practice. Roger, go be acquainted. Thank you for walking us through both the motivation and the mechanics behind our pan-glow and its impact on the rest of the components of Solana. And to our listeners, if you're building an operating in Solana, this is one of these moments where the stack is changing quietly, but fundamentally. So thanks for listening and we'll see you in the next episode. Thanks so much for having us. Thank you.