This is Software Engineering Radio, the podcast for professional developers on the web at sc-radio.net. SCRadio is brought to you by the IEEE Computer Society by IEEE Software Magazine, online at computer.org/software. For Software Engineering Radio, this is Robert Blumman. Today I am joined by Luis Rodriguez. Luis is the CTO of Sygenny, where he focuses on supply chain security. Priorities of Genny, he held management positions and consultant roles at a Junto and other software companies. Luis, welcome to Software Engineering Radio. I am, I appreciate the opportunity to talk in this podcast. Would you like to say anything about your background that I didn't mention? I have worked mainly in the application security space for some 15-20 years, more or less, mainly on the bend or side. I was interested in application security in software security in general, and I worked in later projects in the static analysis for vulnerabilities, for application security vulnerabilities. Also in the SCA space detecting vulnerabilities on dependencies, on type party development. I felt a bit guilty on producing a lot of noise, a lot of vulnerabilities. And in fact, a new project, which is Sygenny, which I started with as a co-founder, intending to correct that problem, to give a better approach to managing vulnerabilities and also the malware. Because not only the passive security problems, but also the active ones, the issues that third parties, bad actors are injecting in the supply chain. This is a, which we are focused on now. Supply chain security has been in the news a lot. We have covered it on software engineering radio a few times. Episode 535 on supply chain attacks, 541 on securing open source supply chains and 606 on third party supply chain risks. Today we will be talking about a celebrated or maybe I should say infamous supply chain attack on the SSH demon through a package called XZ. This attack has been widely reported in the media and I follow this to some extent, although I wouldn't call myself a security researcher. I didn't feel that the media really dived very deeply in depth into this and hopefully we will be doing that today. I want to start out with we will cover what has already been reported on us. Could you give a summary of the attack and then we'll delve more into details. Sure, basically a malicious actor created a command execution backdoor in a popular Linux compression library LCMA XC for short resulting in a backdoor in another popular remote access system, which is open SSH. This was an advance software to play chain attack with obfuscation hiding this behavior from reviewers. It was detected on March 28 this year just one month after it was injected and it was quickly contained. So it did not enter stable distributions, which fortunately was a lucky thing because they imagine if you have a backdoor open in more than 20 million potential open SSH servers hanging on the internet, you have a big problem. What would the attacker have been able to do if this attack had been distributed as I mentioned, there was something like more than 20 million open SSH servers hanging on the internet. That means that a bad actor has access to a arbitrary commands of a selective target. He could break havoc in a lot of places. The problem was contained but it could be much worse that it was at the end. This is the ending of the thing but the problem is that it opened a lot of kind of worms. What happened if this is not the unique backdoor popular system in hiding on the internet, this is the which we are taking into account right now. I have raised a point that there could be other similar attacks that have not been detected. I do want to come back to that question but before we go there, I'd like you to describe how the attack was detected. Post-war developer named Andres Friin, who detected about the performance issue with the SSH server he was testing. He detected that the login took half a second more than usual. He decided to investigate or to perform some internal research on that. First he extracted from the distribution table for the open SSH. He tried to look something there but he found nothing relevant. He then performed some performance testing with the open SSH service. And he found that there was a library which was LCMA library which was acting as a CPU hook. And he made some debugging and he detected that that library was doing something strange which was modifying the system library, modifying the table for functions and injecting a different function which was the pre-autantication function that the SSH uses for validation of the public key presented by the client. He decided to look more in depth in the LMCA tarot. And he found that there was something strange with the build script that was extractive from the testing files which was highly sophisticated and was included in the build system in the final binary for the LCMA library. That means that probably that was some strange thing malware. He did not perform a reverse engineering. He asked for help for others. They found that there was something that tried to open a system called for running the commands passed by the attacker in the payload in the communication payload. So this was clearly a backdoor not for authentication with SSH but for remote code execution. Allowing the attacker to run arbitrary commands using the commands they decide on the vulnerable system. And it was also an attack that it was not replicable and it was gated. That means that only the owner of the private key, the right private key, could make the backdoor run. Otherwise the system worked as normal. It jumped to the normal crypto exchange. So it was a passive backdoor. And this made it impossible for normal detection tools unless the backdoor is really executed. No tool could detect that it was there was something strange on that. But it was amazing how by accident by a performance problem this could be detected by a person which he declared he was not a security analyst. So I think we were lucky on this. I need to discuss also in a bit later in the interview the element of luck that was involved and is that a good way to detect things before we get to that you raised a couple of points that I want to come back to in my reading about this and I only know a little bit about see and I'm not involved with these security issues at all. But what struck me as hard to understand is SSH is the application it uses libraries which is how every program worked compression is a library that it calls upon to do some step in its process. So my question was how could a library that you call insert a back door that affects the behavior at a much higher level of the stack. I think where you begin to answer that is there is a table that points to functions somewhere in the process and that there isn't any restriction that prevents a library from going and changing how this table works. Now that I understand that correctly and can you say more about that. That's a very good question and complex to explain. Open SSH does not link with this library with a live LCMA library. It uses a say live or JIT library but is optional. It is not even linked with many open SSH distributions. So in fact this is an option that is normally not enabled and never works in normal open SSH installations. But in some distros patches were added which enable SSHD to inform the system the service manager in many systems which is system D when it has finished starting up.
SystemD is a common Linux system and LeafSystemD uses the XC library. So the MaliciousLive, the MaliciousLiveLCMA library installs and output hook into the dynamic linker waiting for a function name. I have a small check here. It's a RSA public decrypt. This is the name of the of the function that is used by SSH. To perform the initial authentication logic. So the malware through the system dependency managed to change dynamically the link to this function. So it works with the backdoor. Instead of running through the normal function, it runs through the backdoor. The backdoor, check simply key used to authenticate, is the right one from the attacker. And then diverts the payload to the system called, which runs an arbitrary command in the system. SSHD, the server, the open SSH server, normally runs with high privileges, typically as root. So if you can execute the command as root, you are a king. So this is the reason why they looked for open SSH instead of running in the user space in the compressed library. And it was a very interesting way of without having direct link from SSH to the library. In the library you can manage to affect the third party service upwards. So this is the key of the attack. When I have read about this, it reminded me of, there are other attacks on the process runtime. And there have been efforts over time to restrict things. I don't recall the name of this, but there was an attack where a function would be written into an area of heap and something like what we've been talking about was used to get the execution to point into that function. So there were changes made in the runtime layout where certain areas were data only and you simply could not execute those areas. The process would fail in some other runtimes like Java. You can have quite a high degree of security over what pieces of code have access to what parts of the runtime. Now, I don't know, a cool amount about C. And we'll talk more about remediation later, but to write on this, I wanted to ask, is there been any discussion or is this fixable in such a way to say we can't allow a library to have access to those function pointers? Or is that just how this runtime works and it's not fixable? Absolutely, it's fixable. The problem is that the attackers may be some changes that allow the library to be able to perform this change of the symbol table, allowing to inject the back door in the upper SSHD diamond. And this was fixed, but the idea was to add some kind of performance improvements, allowing to change functions using indirect functions or iPhone, for sure. And this was added before the actual malware were injected by the bad actors. They had a plan. They decided to first allow in the library to perform this dynamic change of the functions in the via system D. And after that, they inserted when they were aware that nobody was looking, they inserted the actual malware. I see, that means they first disabled security feature that would have prevented the attack for completely different reasons. So they said, but their true reason was to prepare the groundwork for step two, which was the attack. Very clever. I think now since we're talking about this might be a good time to talk more about the attacker, which is some kind of entity and is still mysterious. What is known about the attacker? One thing is to have evidence of who could be after the attack. And other is that the activation. Activision is quite difficult now, but the good thing here is that all the changes were done on a public repository, which is a Github version, which has also a copy in the Github Linux system for Github, a mirror. And we can see exactly which actions were done by the bad actors after effect, usually. So there were two commit authors, mainly one named GiaTan. GiaTan is a common Chinese name, but forget about thinking that GiaTan is Chinese probably. And there will also one Hans Jansen. These two guys, they are actually personas, they are actually fake accounts. We're working, accordingly, to perform some activity first. GiaTan, this is a social engineering attack, mainly, because this GiaTan made a neutral or inocuous commits for more than two years since November 2021. He gained confidence from the maintainer, the innocent person here, the last calling. So he was promoted as maintainer during June September 2022. Another commiter, this Hans Jansen, then made two key commits, not Malicious Perse, but preparing the field, as you mentioned, with this iPhone support, which opened the key for the attack. There was also another fake identity, it is a sock puppet, like one named GiaR Comer, Denny Sans. They were using a femoral email systems, they just made some sends and emails to the maintainer, to the problem maintainer, and to the distribution mailing lists asking for reenest or pushing for the Malicious torals to be included in the decision. These pressure emails were done by different persons from the ones who were performing the commits in real repository. Those identities are really fake, using a disposable email, versus a mention for social engineering. And no red flag was opened by the actions by the bad actors, because they worked very caningly without trying to make too much noise. Commiters were even thanked by the poor maintainer for their commits. In retrospect, some dismentainers show concern on the pressure emails received by these fake accounts, for including the versions in the stable release. Probably the best mechanism for avoiding the worst problem was the slowly pace in which Linux distributions work. First, when a new change is done, it enters the unstable or experimental distributions. And only after a full review, it passes to the stable ones. Obviously, you can have an unstable version. I have myself in fact, because I need to be on the on the on the edge of the last versions. But one thing is to have a malware in a normal distribution and unstable distribution. And a different thing, a devilish thing, is to have the malware in a stable distribution, which is much worse. No red flags happened in these two guys, Jatán and Hans Jansen, made the decision changes. And again, there was three stages, the attack. First, the neutral stage, where Jatán made some contributions to the project and gained confidence from the main maintainer. And then they asked the maintainer to be maintainers back up back up maintainers, which is a normal thing when you have done a good work on a project. You can offer yourself as a new maintainer. They deceived the original maintainer. And he approved some some accesses for Jatán, particularly. He made and signed the releases, which were taken by the Linux distributions for adding the library to the to their that the solutions. So one month before the day the attack was was found, he made the final the final step, which was the attack. He added the he changed it in the terrible not not in the source repository, but in the total he made a small change in the bill file in one of the bill files. And with that change, he make the when when the prayer was compiled from the sources, the final binary included the back door. This was the last the last step the attack step, okay, all they have to do is just wait for different distributions to.
to get that this enter, to get the distribution install on the endpoints, and then they could choose which endpoints to attack, because they have the private key needed to perform the back door, to round the back door. - Talking about the attacker, was there any additional forensic work done trying to locate who the attacker might be based on any information contained in the commits? - Yeah, after the fact is quite easy to see exactly which were the commits, there are many commits, but most of them in the first stage were neutral. Only the commits that I mentioned, they changed by that dancing on the enabling the e-funk capability, and the final attacks from Jatán where he uploaded the new releases, the new malicious releases were the key points. Later, there were some bugs. When they use in a tool called bug green because of the way the back door was injected, there were some tools were bending the process. So they managed to fix some of the bugs after the malware was distributed in some unstable in-house distributions. They were checked because they are a number, but not a big number of malicious commits. And after the fact, it was found exactly what which were the things that there were clearly trying to maybe avoid detection, fixing bugs to avoid detection or a performing additional preparation steps for the attack. - Things like time zone IP addresses, were there any clues even indicating where this attack or mine had been located? - We have evidence, but no one question of attribution. First, that Jatán used a VPN for access all the time. He used a Singaporean IP address address. That's not mean that he is in Singapore, of course. Regarding the time zone used in the commits, and the war patterns, we have some ideas. They used a UTC 8 plus 8, which is the Chinese time zone. Obviously, his name is Chinese. So he tried to decide that in fact, in some commits, probably he forgot to change the time zone and he was using UTC plus 2 plus 3, which is due to the summertime change, which corresponds to Eastern Europe and Israel. Also, it doesn't mean that these guys are on Eastern Europe but it's a nice interesting evidence. Okay? And obviously, the working pattern, they work from 9 a.m. to 6 p.m. in that UTC plus 2 time zone, which resembles normal working hours, not nicely working. So all of these, they didn't work on holidays. Hackers, the bad guys, the financial motivated bad guys, normally work on holidays, work whenever they want. And they use a regular pattern. They are not using the working hours pattern. For me, it's work done on the Chinese holidays, for me, it's a clear sign, this guy is not Chinese. Okay? But not on Christmas and in Christianity New Year. So all these evidence accumulates to think that we are probably thinking about a bad actor, not a financial motivated actor, but probably a state-based actor, which was operating on this time zone. I don't know the latitude, I know the longitude, not the latitude, but probably Eastern Europe is a clear candidate. And the most robust evidence comes from the techniques and the pattern itself. Passions, planning, more than two years planning, this is not a financial motivated bad actor, which quickly runs for more money. This is probably other kind of actors. And I think about APTs or state bad actors. The target is also not financially motivated actors. They are trying to get, obviously, they can use a backdoor in SSH, but probably this is more for spyware, for extracting information, sensitive information from certain systems. So I think that this makes the only evidence accumulates on the APT candidate. Okay, the ability to access SSH would be only a step towards some other type of attack. So where did you see them going with this, if they were able to get it in place? Probably, I think they are trying to use a surgical, surgical backdoor is not a PCIe attack. The injector, I mentioned, 23 million APTs potential installations, not all were affected because there was a need of different environmental conditions where you need to meet for the attacker to be able to be executed. But they have a lot of servers where they could use the backdoor. And I think that probably there's a surgical attack to select the targets would be the reason behind the attack. Which ones, I don't know, mainly Microsoft, who knows? Now, SSH logs, whenever somebody authenticates, many security systems would monitor these logs looking for unusual authentications. If it had gone into the wild, could it have been detected through log analysis? - Through logs, no way. Because the backdoor was not doing anything in the authentication phase. It was simply diverting the authentication. And it was running the backdoor, which means you will not find any failure authentication in the logs. You understand? - Yes, got it. - You just got it. No trace on the logs. So no way, it could be detected, but using the different approach. - Now, we've been talking a bit about how it was detected, how it could have been detected. Has there been any discussion in the community about, is there a better way for the community to distinguish between real people and fake entities that are cutouts? - You hit the nail. This is the common problem with open source. In open source, you can even be totally anonymous. If you make good contributions to an open source project, you are welcome. And there is no betting. There is no list in the in general. You can perform your activity from a VPN, high injury identity, totally using names, and you are welcome. This is a problem with open source. Obviously, in the private sector, this is not reasonable, but in open source, that was the way of working up to now. And the change, this is almost impossible. So maybe we have to, for example, the problem here was a small change in the build in the turbo, which was released for the distributions to perform the compiling. And this, so maybe this is the point where it could be detected. But this is also very difficult, because as you know, the turbo has the old source code, but the compilation is quite complex, quite difficult. And it's difficult to have an automated system to understand if there is a difference between the expected output of the compilation and the real one. The first C compiler can follow it, made some hints on this, and how can I trust the C compiler? How can I trust the tools that make other tools? This is a complex problem, and it's an unsolved problem for now. Of course, you can have two different compilations, one system and one with other, and you can compare it, the results are the same, which is in this dual compilation with different build systems, probably you can catch a difference between the two, because for this backdoor to be enabled, you need a particular environment in the build. The problem is that compilation is not repeatable, and you can compare even with the same environment, the same source code, and get different non-binary compliant output. So it's quite difficult to do this. This is for example, for the solar winds, the system they try to change this, and dual one-armatic build, and one non-armatic build, and compare the two outputs to check if there are any difference. So if someone can change source code in the build process, that could be a problem.
be detected. The problem is that it's not easy because of their repeatability problem. He means what you're talking about now is a way that it could have been detected. We talked a little earlier about a degree of luck in that someone who's not a security specialist was doggedly determined to get to the bottom of this performance issue and he knew enough to call in other experts when he saw something that shouldn't be there. You also mentioned that my idea of checking logs is not work. What are some other ways might have been detected or in a more general principle? How could this type of thing be detected? First of all, after the fact detection is easy. I believe there are now a Jada rules for command line tools and a lot of tools for detecting. For example, there are some program capabilities and analyzers. There is one from Changwer, you name BingCapt. There is also a tool named Binally which provides detection rules for this particular malware. The problem is that they cannot detect it before they are instructed to do so. They cannot detect preventively. There are also different kinds of tools which are runtime monitoring tools, which baseline the behavior of the programs and libraries and report on deviations from the normal behavior. They've learned. This might detect exploitation right after it runs. The problem with this backdoor is that unless there is a real backdoor attack, you don't see any change in the normal behavior of the SHD diamond. This makes even for these tools quite difficult to do that. In fact, there was no evidence of a real attack using this backdoor because it was removed quite promptly from the releases. But even if this exploitation happens, these tools may detect them because they will see a system called which was not before in the normal learned behavior. They can't report this as an unusual activity. The problem with this is that you have to instrument the end systems. You don't detect the problem at the source. You detect them at the end systems. You can detect them at the end systems. Obviously, if the end systems report them to a thread modeling tool which gathers information from multiple sites, they could detect this is what happens, for example, with tools like the ones from Krolaustrike and other tool vendors. They could detect that there is an ongoing attack using this SHD. But this is after the malware enters the victims, the end victims. This is not also the perfect thing for detection. Obviously, detecting this bad commits in the source would be the best, but it's quite difficult because you trust the contributors if they are fine and it's quite difficult. If they use their magic, they can hide, for example, in this case, the backdoor was inserted in a test file, in a test compressed file, because this is a compression library. So the binary containing the actual malware, the actual backdoor, was in a test file. And nobody opens a compressed test file to check if there is something strange there. It's quite difficult to detect. You're not giving me a lot of confidence that it would have been rapidly detected. That raises a question that you raised earlier, and I promised to come back to this. Surely people must now be asking how many more similar, not necessarily, attacks on SSH, but equally severe attacks that we don't know about because we didn't have some good luck of a person noticing an anomaly and then unraveling the code changes which led to that. There are more deatons out there. Now, this is for sure. Which projects? And no, there are some projects under review. For example, all the projects that are using I think are being under review, because this was the backdoor, the backdoor's used for this for its attack. But obviously, they can use alert techniques, which are not known for now. And obviously, it's quite difficult to prevent malicious activity when it's done by actors that take the patients in the magnification for the attack. So I have a note asked your question. I don't know. In fact, I don't know if there are more deatons out there working, making their magic on another popular point. This next question we may have covered to some extent, but anything about lessons learned that we have not yet covered? Well, we talk about how hard is to prevent these intentional backdoors because they are done with by knowledge people, not scriptkidis, which was known as scriptkidis when it was young. Some authors pointed out a system D. You know, the system D is the bridge between the two libraries, the SSH library, the SSH executable and the compression library that was attacked. This popular service manager opens a large attack surface, a third party services for backdoor. This is what the bad actor I've used. And a system D has many eyeballs on it, but this, and a skewer library up the chain can poison all the water down the stream. This was a wake up call for other projects that depends on system D to check exactly which kind of backdoors could be inserted using this bridge library, the system D. For example, immediately after the attack was uncovered, there was a full request on the XC library for removing the iPhone support. And this was also a change in system D to avoid using libraries. For example, the pro-nista system D uses XC library and is not related to the attack. But what is the reason for system D to use this library? It's nonsense. So they are skimmering, eliminating all the spurious dependencies to keep a clean, the main important element like system D. This is another learn lesson. And probably the root cause with open source security issues is probably the access trust test. I mentioned that this is the way open source projects was evolving the past. But this is getting a deep review now because obviously an important project needs much more control on which person makes which changes. And this is probably we'll change a bit how open source projects in the Linux ecosystem and other ecosystems are done in the open source community. The problem is how maintainers get remunerated by their hard work. If they are working as volunteers, there are some ideas on how to finance by the private sector, by the government, the open source of efforts. And I think this would be another another with the thing. And use also a different approach. I mentioned that probably the attack could be detected if there was a mechanism to detect problems in the build that lead to a different binary from the one produced by the pure sources. You mentioned that the exploit was in a test file that was included in the final distribution artifact. That also sounds odd to me. If I'm running a Linux system and I want to run SSHD, I should not need any of the test files in my distribution. Am I correct about this? And is that another avenue of improvement? No, no. The problem is that in the terrible you have the source code and the test files because when you run, you run also the tests. When you make the build, you run the test to check that the system build is correct. So obviously the test files need to be there in the source table which is compiled by the distributions. It's difficult if you cannot get rid of the test files. And obviously if the test files are removed, probably the bad guys will find another place to put the malware. So it's not there. I think it's not there. We're getting close to end of time. Something I'd like your opinion on is had this exploit been released in the wild, how would you rank it compared to some of the more well-known open source attacks that have occurred in the last few years? Well, I think it's five, not ten. The CVSS score, when the CBA was opened by RATAD, it was given ten. Ten is apocalyptic. In the past there was some tens also, for example, like well-known log for J issue, receive the ten.
But the problem there is that it was completely open. And there was a lot of every single 2.x version of Love4j was vulnerable. So the problem was huge. Here the problem was much more contained because it affected only a few distributions, not all the distributions, for example, open sus, devian, devian unstable, arcelinux and fedora, fedora from Red Hat. Not small distributions, also cally Linux, cally Linux, which is used by Pentasters, the distribution was choice for Pentasters was affected, interesting. And that for example, devian stable or Ubuntu or many other distributions, commercial ones and so on were not affected. So I gave five. The attack is serious because if it could really get open access, the problem is that as this is a back door that makes no noise, it could left the time between the introduction and the detection good range in the years. Perhaps the risk was high, it was enormous, but it was contained quickly. So for me it's a five, not a 10. Thanks God. And thanks to Andrew Freudian, we will free view for four years. As we come to close, were there any aspects of this issue that we have not covered that you'd like to highlight? I think we covered a lot of things from the detection, from the how the attackers work. They use a social engineering attack. This is the key idea because introducing the malware they need rights to perform changes and to release the distributions. So social engineering is the big deal for many attacks in the supply chain part of the equation. So I think this is interesting to understand how the attackers work, how the game confidence, how they exploit the different, the combination of techniques they used is interesting because with this information we can prevent in the future, similar attacks. I think this is the main benefit for this knowledge, for this attack. I did forget to mention that much of this conversation was based on an article that you wrote on the Zijeni blog. On the issue that will be linked in the show notes. Is there anywhere else you'd like people to find you on the internet? Well, I have a link at the account. If you want, you can reach me by email my email is Luis.
[email protected]. It's been a pleasure speaking with you. Thank you very much for speaking to software engineering radio. Thank you very much. Roguel, the pressure was mine. For software engineering radio, this has been Robert Blumman. Thank you for listening. Thanks for listening to SE Radio, an educational program brought to you by IEEE software magazine. For more about the podcasts including other episodes, visit our website at se-radio.net. To provide feedback you can comment on each episode on the website or reach us on LinkedIn, Facebook, Twitter or through our Slack channel at seradio.slack.com. You can also email us at
[email protected]. This and all other episodes of SE Radio's license under Creative Commons license 2.5. Thanks for listening.
PodcastTranscript AI