EuroHPC JU and the future of supercomputing - Anders Dam Jensen, EuroHPC JU

The reason I think it is very important that we as Europe have the excess scale capability is because it's the next frontier for science. We need to make sure that we have the systems that our scientists need in order to do science so that science can be done in Europe. Welcome to the first episode of Supercomputing in Europe, the podcast where we discuss various interesting tech topics with supercomputing at its core. My name is Apostolos Vasiliades from ENCCS, the EuroCC National Competence Centre in Sweden, and I will be the host for this episode. Today's guest is Anders Dam Jensen, who is the Executive Director of the URHPC Joint Undertaking. We'll talk about the current state of supercomputing in Europe, successes and challenges, its progress and its future. Anders, thanks for joining us. Thank you for having me on this first episode. Can you introduce yourself, give us a little bit of your background and how you ended up as the Director of the URHPC, J.U. Oh, a windy road. But anyway, yeah, I mean, I said I'm Anders Jensen, I'm Danish, basically how did I end up here? Well, I circled back to what I did at university. I did play with clusters back in the university days, then I got into chip design and was there for about 10 years, and then I got sidetracked and became the Director of IT for a significant airline in Luxembourg. It's not of general public knowledge, but there's actually a very large airline in Luxembourg called Cargolux. It's all about cargo aircrafts and flying cargo around the world. It's actually quite fascinating because one doesn't think of Luxembourg as a hub, at least from a passenger point of view. But from a cargo point of view, it is actually a hub because there are no passengers. So there's a very large airport that supports cargo and therefore also a very large airline that hooks up with trucks that takes the cargo to their respective destinations around the world. So I was there for 11 years and then it was time to try something new. Long story short, I ended up in Brussels working for NATO for nine years where I was basically the lens CIO. And after being nine years in a job where you only supposed to be for six years, I was kindly told it's time to move on. And here I am. So I was very fortunate that the GU was standing up and needed an executive director, and I was chosen for that job. So that's how I got back into something I did a long time ago. In our line of work, we hear a lot of terms like supercomputing, HPC. Tell us what it's all about in a way, maybe not a child, but someone non-expert could understand. Well, I'm sure there are as many definitions of this as there are people involved in it. So I'd like to think of supercomputers or HPC as essentially, if we look at it technically, it's when we scale beyond one node. One thing is to program your solution for running on one node and scaling that node up in itself, taking the most powerful CPU, maybe even adding a GPU. But where it gets really complicated is when we need to go beyond when that thing can't do it anymore, and we need multiple nodes with all the complexity that that leads to. So for me, the supercomputing aspect is really we're talking about devices or machines and ecosystems where we're doing things that could not be done on any normal machine. It's not only the machine, it's the whole thing that needs to be adapted in order to take advantage of this technology. And when we look at some of the monsters we've now managed to procure for Europe, we're talking about vast machines and unless you have the knowledge on how to harvest that power, it's not as simple as writing your program for a normal machine. So we're sitting here in Antwerp in Belgium for the URHPC Summit, which brings together people that are interested in working in supercomputing around Europe. Can you say a few words? What is URHPC? So yeah, definitely, URHPC was created, first of all, we are an EU body, part of our budget is funded by the Union. Now we are a special type of body, we're what's called a joint undertaking, which means that we do all, everything we do, we do jointly. And jointly in this case is with participating states. So when we build a supercomputing, we buy a supercomputer, the EU will, you can say, rule of thumb we will pay half and someone else will pay the other half and that someone else will be the one that's hosting it and operating it. Now why were we created? We were created because the Europe in general came to the realization that there had been a significant under-investment in supercomputers across Europe and it was evident from the top 500 list we were dropping off and it led to a lot of very good science either not being able to be done or researchers having to go outside of Europe in order to get their research done. And basically there was a significant initiative within the commission in around trying to rectify this together with the member states which at the end led to forming the joint undertaking and with the mission to try to, with the mission to rectify these shortcomings. So the joint undertaking we have, you can say, technically speaking we have more pillars because that's the way it is but if we look at it broadly we have three main areas of focus. We have infrastructure where we've been given the mission to procure infrastructure and make it available to the researchers in Europe and the way again that works is that we select a hosting site which will usually be a well-known supercomputer center and the reason I say well-known is because they need to know what they're doing at the end of the day. That doesn't mean that the ones we don't know may not know but anyway you get what I mean. So we have these significant supercomputing centers across Europe that's been doing supercomputing definitely before the joint undertaking was created. And we go out, we select a hosting site to host a EuroHPC supercomputer and then we procure the supercomputer with them so that we make sure that the computer fits into their thinking about the specifications we work out in conjunction with the selected supercomputing center. And then the important thing is that because we've usually paid half of it and the other part is paid by the consortium of countries that may be behind the machine, we control, we by we I mean the joint undertaking, we control half of the compute time. And what the joint undertaking then does is we make that time available to any researcher across Europe. So basically the rules for the initial machines as they were funded by Horizon 2020, well if you're from a country that's associated with Horizon 2020, you can apply for access. So you don't need to be in Finland to apply for access to Lumi for instance or in Italy to apply for access for Leonardo. If you're applying for the JU time, any country, any researcher from a country that's associated with Horizon 2020 is able to apply. So that's another part of our mission is that we have a peer review team that has been set up so we get applications in for all of this compute time that we now have available. And we look for excellence in the projects, selects the projects and then we give them access to the machines. So that's one major part of what we're doing. Then we have the other part, which not only did Europe realize that we'd fallen behind on supercomputing capacity, but they also came to the realization that we'd fallen behind on knowledge and this is knowledge all the way down from how to produce a supercomputer to also how to use them. So we have quite an ambitious research agenda where we are, as we like to say, we're investing in the whole stack. We have projects that are creating CPUs for future supercomputers. We have projects to do, we've just closed the project and we're evaluating the project for an interconnect because these are key technologies that we currently do not have European technologies for. So we've invested in those and if we move up, of course, then we go into the software side where also if we look back at in time, there's been a number of projects looking at both the system software as well as moving up into the applications. And last but not least, we have the whole skill side where the JU is investing in, for instance, a master's program for HPC and this is quite cool here at the network. We now have our students, basically, we're using them as ambassadors and they're helping the people here and running our demo lab. So that's another significant activity of the JU to try and get these abilities back into Europe and progress the supply chain, the European supply chain. Of course, the flagship in this at this point in time is what we, the significant investments that went into the European Process Initiative, we will now see as part of the Jupiter supercomputer. So it's a fantastic achievement. It takes time to do these things, but it's really good to see when they come to fruition. So I would say, so that's in a nutshell what we're trying to do. Now, I do need to point out that while we talk about supercomputers, what we've also been asked to do, of course, is to do the same thing for quantum. So we are in the process also of procuring quantum computers. We have four machines that where the procurement has started, there will be two more following. And what we've done for the quantum computers, because let's face it, we are, it's a more, it's a technology that's more in its infancy. And there are, we are, I think we're all the way, there are multiple ways to do a quantum computer and no one's settled in on what is the best. So we've really looked for diversity in technology there and we're buying, the plan is a total of six quantum computers in the initial setup, but all based on different technologies. And the idea really being, let's get them out there so the European users can get to know what it means to run and use a quantum computer so we can all get wiser. And we hopefully can also get wiser on which technology is better for, and maybe we'll get to the situation where one technology is better for one application, another technology is better for another. I mean, this, this is all, this is also to be seen. We'll come back to a quantum computing deal. But let's talk about those supercomputing systems that are already operational. What are their specs? Maybe differences one another and does each one serve a different purpose? So again, there are, there are a lot of opinions out there about what's the best architecture and I think there's as many opinions as there are people involved in the, what I think is important to, to just make clear is the, what ends up on the floor is the result of, of, of an open competition and, and we go out and we, and we set specs in our non-technical way to, so that it's open. So we're looking for performance in, in, in various benchmarks. Of course, I mean, Limpac has been the, is the one that we tout when, when, for the top part, but there are a number of other benchmarks. So each, each hosting site, when we approach them, they, they work out, you know, for their, for what their field of, the fields of science that they're looking to, that they're looking to, to use the machine for, which benchmarks fits that. And that's a major, that's a major component of, of, of our procurements. And that results in various architectures out there. Each, you can say that each, each of them won because of the, of, of, of, of, of how they promise to, to meet the specs. And, and so we, and I think actually we, we have, we have a diverse set of architectures, which I also think is, is a strength. I mean, it's not, it may not be initially helpful that if you need to move from one machine to another, you, you may need to, to do something to your application, but it does mean that we, we also have the ability to get, to get wise on, on all of the, the different technologies that are out there. So, you know, in Lumi, we have the, the AMD GPUs and, and in Leonardo, we have the Nvidia GPUs. And in Jupiter, we will have the new Grace Hopper arm based CPUs, for instance. So this gives us a diversity where, where we will also, as, as the European user community, we will get to know which, which architectures might be better for one situation over another. Talking about Jupiter, would like us to talk about the expansion at the moment, how many supercomputers are there operational? So what we have is, we have the, as we said, we had the initial eight machines that was foreseen when, when the JU was, was, was first stood up. All of those are inaugurated and after the inauguration. So what we've usually done is we've had the LIMPAC run and so on, but, but there's always a little phase in phasing them in. And so, so we go through a phase where we let, where we let users loose on them for, for a while. And sometimes in that initial time, of course, there's still tweaks that's being done. So if we look at it, I mean, right now, the, the, the Kallion in Portugal as well as Miroslav five in, in Spain, those are the ones where we're running in at the moment. And, and, and once, once that's it, then all of them are completely available to the, to, to the scientists across Europe. And there are eight or there's eight, this, there's eight in the original bunch. Yeah. And then of course we've, we've moved on. So, I mean, you mentioned it, we, we spoke about Jupiter briefly. That's, of course, what's so special about Jupiter? Jupiter, I mean, what's not special about Jupiter? There are a number of things that are special about Jupiter. First of all, it's going to be Europe's first exoscale machine. And, and. Kluwassin, what, what this entails? Well, I mean, essentially it means, it means being able to do a billion, billion calculations per second. You know, it's, it's, it's completely, if, if we look back at where we were before we started procuring machines for the, for the JU. I mean, it's, it's astonishing that we are, that Europe has been able to, to, to get equipment like this. So, so one thing is that, that it's, that it's, I mean, this is from a JU achievement point of view or, or, or mission point of view. That is, that is, that is very important, you know, and as I like to say, we're not getting an exoscale because the US has one, but we're also getting an exoscale because the US has one. You know, we also want one because it's important for science. It's really important to be able to go to that next tier and, and see, you know, the, the applications that are able to take advantage of such a vast amount of compute power. And of course it comes with all the complications that exoscale comes with in and around the, the, the interconnect between the nodes and so on. So, it's going to be really, really interesting to see as this thing unfolds and, and as we, we get it ready for the users what, how, how this is going to play out. But, but there are a number of other aspects of, of Jupiter that's, that's really interesting. And, and, and here I just have to, I just have to complement the, the hosting entity that was, that was chosen. I mean, in Eulish, basically, Thomas Lippert and, and the team around him were very quick to, to also embrace the idea that we, that they would like to see some European technology in there. And, and we couldn't have done it without, without them, I mean, it was, it was very easy for me to agree to, but, but, but I'm very, very grateful that, that Eulish was, was very receptive. Well, not only receptive also actually initiated the discussion on it. And this is why we've been able to announce that we are going to have a significant petition in, in Jupiter that is based on the, on, on the Cypheral CPU, which is the results. It's the tangible results of the, of the European Process Initiative. And, and of course, since it is Eulish, that's the, that's the hosting entity. I mean, of course they've also, and again, this is where I say we, we work with the each host, hosting entity on, on the, on the specs when we go out. And of course, one of the things that, that Eulish and his team really, really also wanted was the concept of the modular supercomputer, which is very much their brainchild, and wanted to make sure that, that, that was, that that was represented in Jupiter, so that we could also leverage the, all the work that had been done in the, in the different projects that led to that, which are European funded projects. So, so, so what we're having with Jupiter is a modular machine, and, and what we, what we refer to as, as the cluster partition in it will be based on, on the Cypheral CPU. The Cypheral CPU was not chosen just because it's European, but it's, it's nice that it's all, but because it, it has, it promises some memory bandwidth that goes beyond what, what we can get with other CPUs. And that's the technical, you know, the technical advantage of this, of this CPU. So again, it's going to be really great to see this thing come to fruition and see how we, how we use it. So let's zoom out and talk about Exascale a bit. In your view, why is it so important that we reach Exascale, not only Europe, but generally in the world, but also what are the challenges to towards it? So, so the reason I think it is, it is very important that we as Europe have the Exascale capability is because it's, it's the next frontier for science. So, so again, we're back to why did we start the joint undertaking and the investments? Well, there was an underinvestment in, in supercomputers. There was science that was not, that could not be done in Europe. And therefore scientists had to go outside of Europe in order to get access to the systems that they needed for their science. And so again, if we give up on, if we stop and say, look, we'll let the rest of the world do, do Exascale, well, then, then we're going to be in the same situation. We need to make sure that we have the systems that our scientists needs in order to do science so that science can be done in Europe. And that's not to say that it's necessarily always a bad thing that it's not being done in Europe, but there are certain things we are with there's been research being done on where it's just really, it's just really helpful that it's that we're also able to do it in Europe and in some cases that we are doing it in Europe. You talked about people being able to access those supercomputers through access calls and so on. I want to ask to you to explain who can access those supercomputers and what are the criteria to get access? So I mean, we have for the details, there's our access policy, which I would point to and anyone is welcome to download from our website. Essentially, what we do is the time that we control on each of the supercomputers. We have open calls for them. So there are, and you'll find the details, but you apply for, well, first of all, you should and can apply for what we call development access or benchmark access, which are small chunks of resources, but not that small. And they allow you to, there's not, the process to get those accesses are more just a technical validation that you actually, I hate to say it, but you know what you're doing before we let you loose on it, but then the idea is that you use those to prove that your application can take advantage of the machine and can scale to what you're doing. And then with on the back of that, you would apply using either regular access or extreme scale access, which basically just is differentiated in the amount of resources you can get access to. And of course, the more resources you ask for, the more rigorous we are in the process. Now I say we, that's not, that's not we, the J.U. as such, we have a committee that works with us behind the scenes and you can equate this to, if you're applying for a grant or for a project, well, this is the same thing. We go through a very similar process of having external reviewers that evaluates each application, ranks them. And then we have a fantastic committee that works with us and what they've been really good at doing so far, since they come from the scientific community, they know the scientific community, they've been really good at trying to accommodate as many projects as possible. So knowing that, you know, some projects may apply for X resources, well, if they end up getting 90% of it, because of the knowledge they have, they believe the science can still be done, but that might allow another project to also be able to move forward. So as the resources, especially as we were building up, were scarce, they've been really good at helping the J.U. get as many users on the systems as possible. Now back to your fundamental question, who can access? Well, again, the initial eight machines are funded by Horizon Europe, and therefore the countries that put money into, oh, no, sorry, by Horizon 2020, and so the countries that put in money into Horizon 2020, well, they are, applicants from there are welcome to apply. So whatever entity in those countries can in principle apply, whether it is industry or academia? Yes, yes. I mean, at the end of the day, when it gets to industry, there are separate rules for SMEs and so on. At the end of the day, we're talking about open science, so you have to be willing to share your results as the access is free. And we do have some abilities on the SMEs side to be more lenient. You can find more details on that in the Access Policy, and it's actually with the introduction now of AI, it's being, well, it's being further reviewed with the aim there that we firmly believe that there are some SMEs that could, that will never be able to afford such an infrastructure on their own, and therefore it will be, it is helpful to get them, give them access. Since you mentioned AI, historically, supercomputers have been used by physical science like physics and chemistry for their complex simulations. Do you see a shift towards other disciplines like AI, data science? Definitely. I mean, there's a clear shift. Well, a shift may be, maybe actually the wrong word, we're getting a new user community. It means there are more users fighting for the same resources at this point in time. But rest assured, we will also continue to invest because it means there's a higher demand. But we are being, we have this new user community which also, especially has a, it shows a lot of interesting promise, but it's a very different user community from our well-known, well-established scientific community. What I mean by that is the established scientific community, they've been using supercomputers for, I was going to say decades. They know what it means to run their applications on it. They know what it means to scale in that environment, and they know how the supercomputer, the traditional supercomputer works with the queuing systems and so on. What we have with the large part of the AI community is a community that has more grown up from having a very large workstation with as many GPUs as they had PCIe slots stuffed in there, and realizing that if they could get access to even more, well, they can do even more. And this is where it gets interesting. But we're back to the, they need help because it is fundamentally different when you are scaling beyond your one node. And this is what we're seeing with some of these projects, is they have great ideas, they have clearly maxed out what they can do on one node. But going beyond is not straightforward. So we're trying to address that. The JU was, we're setting up an AI support center, and we've also done something very specific in around a support to SMEs in order to try to give them the necessary help in order to utilize the supercomputers to their potential. And this also, I mean, because they're growing out of that workstation, I'm generalizing a bit because there's, of course, also some AI users that fully well knows what it means to run on a supercomputer. But we have this large amount of new users that isn't in the situation that they came from that workstation. That also means they're not used to having to queue up in a slum system or the likes. So all of this is a challenge for the environment in general, because we have the supercomputers that are configured for good reasons in the way they are. And because we need to maximize the resource utilization and incomes, a user community that is used to a much more, you know, I press the button, something happens. And we need to find a way to accommodate both. And there's no question that we are, we're going to be moving in that direction. We've already said we're establishing these support centers. And of course, the big thing, which I cannot speak to here, because it's in the hands of the commission as well as member states. But it's no secret, because it's been announced, that there will, most likely, there will be an update to the URHPC regulation in which this user community will be brought into the JU. And we're going to have to then define this whole support structure around it. Apart from AI, do you see any other trends in supercomputing lately? That's a good question. I think we still, as the JU, with the bigger machines coming online in the last year, I think we still have work to do to bring all of the applications on board and the different application communities. We've seen, and just yesterday here at the conference, we saw four projects that had very different scientific backgrounds that was showing off results, which is really gratifying to see that the users have embraced the machines and they're starting to get good results out of it. So I think you say that's not answering the trend question as such, but I think the machines that we're providing with the vast GPU capacity and so on is still something that the user community now needs to spend a little time getting to. And then you can say on the technical side, well, then the question is what's next? I mean, with Jupyter, we've bought the latest and greatest CPU GPU for the booster partition. And I'm sure as the plot unfolds on some of the other machines that we're going to be buying, we're going to be seeing whether the AI capabilities will be something -- I'm sure it will be something -- that we will be looking more at in the technical specs with the view that if we're selecting machines that are, what should we say, catering for the AI community, it may be helpful to also select hardware that are more catering for that than for the traditional scientific workloads. I think in all fairness, I think that that would have two advantages, first of all, if we were to get equipment in hardware and that caters for the -- then maybe we'll get more bang for the buck. But equally, we'll also free up some of the other machines, because if we put the AI workloads then there, then our scientists can benefit from the machines that are hardcore FP64 supercomputers. Speaking of hardware, coming back to quantum computing, can you explain the difference between classical and quantum computing without being too technical? So you can rest assured that I'm not the one that's going to get the technical on the quantum. I think it's absolutely fascinating that we've been given this task. I think it's really, really interesting. But it's also, I have to say, it's a field where I've had to do some reading. And I'm not fully well-worshed in all aspects of it. But the fact that the JU has now been asked to buy -- as I like to say, we're buying the toys so we can make the toys available to these user communities so that we all get wiser on quantum. And the reason I refer to them as toys is not to diminish them, but I think it's in any way shape or form. It's more to highlight that what we're doing is we're buying six very different technologies in order to find out what is best around each of them. And I am not going to sit here and pretend that I'm the expert in each of the six technologies. But what I have realized is quantum promises to be very good at things that the traditional computer is very bad at and vice versa. So from that point of view, it's a match made in heaven and when we get all of this to work. Which is why we've invested heavily in it. So the six quantum computers that we're getting in, each with different technologies, are being hosted together with six supercomputers. Because the firm belief is that pairing them up will be an advantage so that we can offload those part of the problems that the quantum computer promises to be really good at. Now, there's a lot of things that still needs to be done. And that's why we're also funding projects to do this integration work. Because all of this is the concept and an idea that is being funded because we firmly believe this has legs, this will lead to some, but it's not as mature as HPC. So we're really leading the way. We're doing something that's not necessarily the rest of the world is doing. And Europe is being seen also by some of our collaborators outside of Europe as leading the way. As making a bold move, we're investing in the infrastructure so that we can all across the board get wiser on quantum. So seeing all this growth, you mentioned Exascale, scaling software for different architectures and computers, quantum computing. I would assume there is a need for specialists and generally domain experts in multiple relevant fields. Would you say that there are proposed actions by the European Commission or other European institutions to tackle potential competence shortages? So there's a lot of focus on that, both from the Commission, but also from the JU, but more importantly also from our governing board. So participating states are also very tuned into the fact that we need to do something in and around skills. So as I said, we have, of course, our flagship program at the moment is the master's program, but we're trying to address skills and education at all levels. So it's not just the master's program. We like talking about it because it is a significant program and the students are going through it now. We're on the second year. It's really gratifying to see those students here in Antwerp showing off the different supercomputers. So they have been put in charge together with the hosting entities of our demo lab, for instance. They are showing code that they have done on our supercomputers to the delegates at the conference. And of course, that allows them also to interact with the delegates who hopefully will wish to bring them on. And they are in dire need for skillsets. But we also, we have a project that's looking at traineeships. So again, the next tier where we're saying, look, someone with the right qualifications come in, work next to a supercomputer for a while, get to know it, and so on. And then the call that, I'm very much looking forward to us bringing out, we're still discussing the details of it, but that's the concept of the EuroHPC Virtual Academy where we're also trying to pull together various sets of training. So there's a one-stop shop or whatever we wish to call it. Coming to the end of our discussion, let me ask you one final thing. If you would choose an interesting fact about supercomputing during the last years, what would be, what would your focus be on? Oh, I should have thought about that one. There are so many interesting things. I think the whole shift to GPUs is interesting in also in all of its complexity. And I think the realisation that we are all of a sudden memory bandwidth constrained rather than compute power constrained is what that means to applications and especially the interconnect. I think it's a really interesting shift and where applications need to learn how to scale and take advantage of this new infrastructure. And my background is computer science. So for me, I'm the geek that thinks that that's really interesting where there's a new programming paradigm that needs to be harnished and then you see everything that happens around it, how the GPU gets to have a more and more dominating role also in what it's able to do, sending its own MPI messages and so on. I mean, who would have thought? So I think that whole area and how we learn to tame that is fascinating. Anders, thank you so much for being with us. Thank you for having me. This episode was brought to you by ENCCS, the University National Competence Centre in Sweden, which is hosted at Rice Research Institutes of Sweden and Linköppen University. We give supercomputing training and support to Businesses Academia in Public Administration for free. For more information, please visit our website, ENCCS.SE. The supercomputing in Europe podcast is made possible with funding from the European Commission and specifically from Euro-HPC Joint Undertaking, the joint initiative between the EU, European countries and private partners to develop a world-class supercomputing ecosystem in Europe. The podcast is sustained by the UCC project, a network of national competence centres in Europe to provide supercomputing training and support for industry, academia and public administration for free. If you want to learn more or contact your local competence centre, please visit ucc-axis.eu.