Magic Packets & Stealth Backdoors: The Art of Detection Engineering

[MUSIC PLAYING] You're listening to Discarded, Tales from the Threat Research Trenches, a podcast by Proof Point for security practitioners. Each episode, you'll hear from security researchers, malware analysts, threat hunters, and more, as we dive into once going on in the world of cyber attacks and how defenders safeguard us from threats. Let's get into the show. Hello to all our cyber-daffodils. It is springtime here in the world where I live, and I hope everyone is enjoying warmer weather or cooler weather, depending on where you are tuning into the podcast from. I am your host, Celina Larson. You are listening to Discarded, Tales from the Trenches, a podcast by Proof Point. I am here today with my co-host Tim Krumper and his two dogs. Always. I'm here with Senior Threat Detection Engineer, Stewart. Welcome, Stewart. Well, have you. Hello. Thanks for welcoming. I am so delighted. Welcome to the podcast. You are new on our podcast. And so why don't you share with our listeners who you are? What do you do? Well, as you mentioned, I'm Senior Threat Detection Engineer over on the merging threats team. I joined Proof Point just a little over a year ago. I've been enjoying writing sigs for both the ET community and for our customers. Well, welcome. Welcome. I have to say, one of the highlights that I have with you, Stewart, is that we were wearing matching corduroy when we hung out for the first time. Yeah. Yeah. Good name. Yeah, I had my overalls on. It was great. It was my fancy farmer style. I got it. It's great. I love it. I feel like we have a few fancy farmers here at Proof Point. Even Sarah, definitely five in that way. And I somehow matched with like four different team members at our onsite on different days. I don't know what it is. I'm a chameleon, clearly. But one thing I didn't prepare for this podcast was I did not get my nails done. I feel really-- I wanted to match your energy going in. Well, yes, there's always time for Pokemon nails. I'm just saying, next time you got to bring it. All right. Next time we all need to dress as fancy farmers, because that would be fantastic. Yeah. [LAUGHTER] So all right. Let's dive into some stuff. So one of the really cool things that you do here at Proof Point is you are taking a look at exploits and backdoors. And recently, there have been some spyware leaks that have kind of come out like source code that's been leaked or tooling that's been leaked. And you really dive into that type of malware, quite a bit. And I know that you have done some research on historic backdoors as well. So I'm kind of curious. What are you currently working on? What do you find really interesting about some of the more sort of sophisticated backdoors that we see in spyware espionage? Yeah. So one of the things I really like is magic packets and port knocking that backdoors will implement. There's a few that do really well. There's a few that are pretty prolific. Some have even-- aren't any use anymore, but you find out later down the road, they've been in use for over a decade. And it's just now being talked about. For example, VPF door actually just last week, Rapid7 put out an article of a bunch of variants that they had been observing. And it's been really interesting. There's a lot of different ways that they've been changing up their tactics. But for the most part, it still is the same method. They'll use VPF, which is Berkeley packet filtering. We can't just pass up magic packets and port knocking. What are those? There's some magical. Yeah, magic bytes, magic packets are a sequence of hex bytes usually or a secret order that the backdoor is expecting to find. So from the client side or from the thread actor side, they will organize a packet in a way that the backdoor is going to filter and activate upon receiving. So this method is derived from Wake On Land, which was pretty big in the '90s to remotely access computers and wake them up. The concept of magic packet triggers, inspired covert activation mechanisms. Well, that stuff is not directly related to malware. It definitely inspired a lot of techniques. And in port knocking is similar in the sense that it's expecting a series of ports and a sequence that it connects to before it can finally make an established connection between the client, the thread actor, and the backdoor, which is the server in this case. So it's like a secret knock at like a speakeasy or something, right? Like a haircut, two bits. It's like, that's it. You can get in. You're good. You're allowed. So is this method of the stuff that you're looking at? It seems like this is typically in more sophisticated malware samples that have a little bit more ability to fly under the radar. They're trying to be a little bit more secretive, likely more espionage focus is that what you're-- that's what you're looking at. Yeah. And targets usually are usually like routers, things that are supposed to internet less likely web applications or devices hosting web apps and stuff like that. So-- Internet of things. Yes. Yes. See, talk about a lot about those. So you mentioned the Rabbits 7 report on D.P. Eftor, which was kind of updated research from previously published information about that malware. Can you talk about why you thought that research was so interesting? And then in your role on emerging threats, which of course is creating network detections, I guess, IPS signatures based off of open source intelligence, essentially, how did you incorporate some of that research into writing detections for our platform? So what I find so interesting about is that it's challenging. It's challenging in the fact that you have to both marry performance and accuracy. And stuff like this is really hard to thread the needle on. And so because what you're working with is so small and, for example, when I write a rule for Saracotter or Snord, if I have content matches that are really short, that can impact performance pretty heavily and so magic bites, magic packets, they're really small. They're a small sequence of hex. And so trying to coordinate the matching of that with packets that are more suspicious, they don't match normal traffic is the best way we can match or create a rule that is both performance and accurate. So I find it challenging in the sense that it may not be perfect for everybody's environment. But what we do is we try to put out rules that will work in a lot of environments. Some environments utilize our rules sets through the sandbox. Some do a little bit more work and they do TLS inspection. So they decrypt everything and then re-encrypted before sending into where it needs to. So all that matters, but the magic packets, they're a little bit more difficult to work with, but that's what's so interesting. - So based off of Tim's metaphor, you basically have to puzzle out what the secret knock is and then be able to copy that and incorporate it into rules. So they effectively fire and can detect the secret knock. - Exactly. - And not be too noisy that you're going to annoy analysts that are looking at decent go. Okay, I'm getting a lot of these. - Yeah, do you knock too much, if you knock too much on the speak easy door, they won't let you in. That's the point of a speak easy. - Exactly. - You don't have to be quiet. - Yeah, as a former stock analyst, I tried to envision myself in that role again. And when I write these rules, I do not want to make their life harder. I want to make it easier. And yeah, it's something that I carry with me is stock life, stocking the trenches. - That's great, that's great. 'Cause we need people to think about those things. It was former stock analyst myself. Like you got to have folks that understand what that job is like. 'Cause it's quite overwhelming at times. - Right. - So it's really important that you have to get these signatures right because one of the things about BPM door, according to Rapid7, they call it, quote, "sleeper cells" in the backbone. So it's very important. It's targeting telecom networks. They are very strategic espionage targets and telecommunications infrastructure is a very hot target right now. And we've seen a lot of samples recently. And certainly from China targeting this sort of strategic area for them. And so of course, when you're writing these very complicated rules, You have to make sure that there are two. and very, very effective and don't, you know, false positive blow up the sock because it is a very serious malware. And, you know, if you're targeted by this, you want to make sure that it's 100% correct because of the capabilities and the targeting and the association with espionage. Yeah, what kind of hurdles did you have to overcome that? Well, in older variants, like the 2023 samples, there was a specific TCP sequence and acknowledgement numbers in the TCP header. And that in itself is very unique, but at the same time, not impossible to, to match on, to replicate. So finding, finding a way that, both says, hey, this is going to match on this particular sequence and acknowledgement. But what, what else within that packet would make it look like it's not normal TCP traffic in that sense? Because you need the context, right? You can't just say, hey, here's the, the pattern. Because you need the context to go with that in order to make that more. Yeah, there's 2025 variant where they started at the beginning of the payload, it would start with an X. And then it would be followed by the IP, delimited by a colon and then the port, which would be the callback. But what helped in this case was it was sent over port 53, which then kind of begs the difference. Why, why are you having a potential DNS traffic coming inbound to your environment and then validating that it doesn't have the same structure as a DNS packet would have. Yes, because you wouldn't typically expect a DNS packet to be inbound on a regular network. Yes, that's the. And I think for classic BPF door, there were a bite sequence, there were a few bite sequences, one for TCP and a separate one for UDP and ICMP. But the, the interesting part or at least the, the part that kind of like makes it stand out a bit is that this payload was sent in a. TCP sin packet so no established connection was made yet and normally sin packets don't usually have a payload. So that kind of made it a little bit more, you know, stand out. So it's like a secret hand. Yeah, too. So yeah, nice. Nice. We got secret knocks secret hand shakes. That's fantastic. We need a secret third thing. Yes. Sinister third thing. So it's really interesting how you talk about all the different things that are required to go into detections because it is a little bit like a fingerprint, right? So you're, you're creating something that you'd be able to match in much the same way as like detectives look at the fingerprint on a door in a burglary. Because you're like, okay, I have to match it exactly with this person or this malware as the case may be. And so you're kind of writing that from scratch, basically trying to figure that out. And it's also interesting to how the fingerprints change with the malware. So unlike some more basic commodity types of malware stuff that's, you know, easily accessible on GitHub, things like, you know, information, dealers like Agent Tussle have been around forever. Those are fairly easy to say on the barely static. You get a little bit more sophisticated information, dealers that use different types of, of crypt, like, cryptying or cryptors and different ways of trying to like encode various different data. So kind of like the, the challenges increase as the malware gets better. But what is pretty notable is there's always sort of this like set or type of of malware that is going to be really challenging to say on that has these. And like magical little characteristics and you were talking before we started the podcast on, you're even looking at historic data too. Like so you're like, like investigating how has malware previously used some of these techniques. What can we learn from that? So can you talk a little bit about it? I think you mentioned shadow brokers as an example. I sure did. I find it really interesting and, you know, it is available on GitHub. And it was nice to pull down and take a look at how it tries to do the same thing with a magic packet. Same with BPF door where you're getting a sin packet with a payload size stands out. But what they did was really interesting as they implemented a custom BPF virtual machine, which was used to validate the packet before it decrypted the payload that it came with. So, so this to the start it sends a packet, a really small packet. It's like 136 bytes. The structure of it was there was a trigger value followed by the encrypted payload followed by the size followed by a reserve bytes and then followed by a checksum. And so what was what was interesting about how they they set up the sequence is you had a static value you would XOR against the checksum. You would take another static value XOR that against the size. And then the checksum value would equal the trigger value. And then the size value was checked to see what the size of the the total payload would be. It was really cool. The one part I did miss was before they XOR those values, they would switch the indianness of the the bytes. And so normally the Indian network Indian would be big. And then they would swap it. So as you see this the hex sequence of two bytes, you might have like B E E F come over the wire, but they would swap it. And it would be E F B E. And then they would XOR that and then that value would be checked against another value. The final value. So with those two in mind, that was the that was the first stage before decrypting the encrypted payload. And then they would call back to their to the client that was reaching out. So that was really cool. The structure of it, you would take this payload and all you had all you had to do was do a little bit of complex, you know, bytes, swapping and and XORing, I found it really cool. It was really like everything is kind of located in this package just to check and all you had to do was do the check yourself. Unfortunately, I got part way through creating a nice sericada, a rule for ideas to to match on it. But because of limitations within engine couldn't complete that that process, but that that kind of started a nice little like, hey, OISF. Let's like let's expand some of the features of sericada so that we can do some more complex math or complex comparisons when we see stuff like this. So I'm curious. So you're talking about the shadow brokers, stuff that's on GitHub and on the back door that they were using, but it's been around since what like 2016. And so it's like 10 years old, but you don't really see this much with malware. And I'm curious if you had, I mean, well, first of all, this is this might be a little bit of a spicy question because you're looking at the detections, you're the data, you're the data guy. But I'm curious like, is it that we're just not seeing the malware because we're not catching it? Or do you think that it's just like so technically difficult to implement this type of malware capability that there's just not a lot of people that would bother doing it that way. I would say my money is on that we're not seeing it. I mean, who's thinking of of that of that that sequence. I mean, you could probably think of a really cool sequence to check something, a certain type of knock, you know, 10 things of a different knock. I think of a different knock. And I doubt we'll all have the same knock and to expect one of us to know without without understanding or having heard it once before. Would we be able to guess it on the first try. So, but it does it, you know, like going through these situations in it, like, you know, experimenting and and testing out and seeing how it's done. It gives us like that one extra step to knowing maybe, maybe we can catch up to catch something like this in the future. Maybe we can, you know, surprisingly, someone people love to reuse code, you know, and re-implement things. And it's easy. But also, you know, if we can, we can catch it. That's awesome. And if we just learn a little bit more, that's awesome too. From listen to you talk, it kind of sounds like these, these malware developers are they kind of have a similar job. you, right? They have to listen for these connections. And then you also have to look for the same kind of things to kind of filter out the noise. So like, you kind of have a similar job there. But, you know, not all these are so quite so dynamic, right? We have some that are more on the static side. How do you approach making IDS sigs for those? Do you get inspiration from like other signatures? Do you have like a mood board or maybe like a Pinterest kind of thing you set up for a variety of IDS? Well, what do you do? What's your process? Yeah. Yeah. I do have a mood board. I do have a vision board. And it is based on Yara rules and host-based detections, sim rules, things like that because I I come from, you know, I did a lot of software where you are looking at Sims, you are looking at all types of logs and doing some research into malware and what, you know, what it's what it's going to do. What is the components that make up that malware? I have this vision that most Yara rules that are written for detecting binaries statically, like rather than in memory because this wouldn't work in the same sense. We could almost implement the same technique in IDS. There are some limitations. There's size limitations. There's only so much that we can inspect at a given time. A stream can only be so long that we can we can inspect, but there are certain ways around it. I do look towards Yara rules for inspiration for guidance. There's a lot of great researchers out there that put out a really good content and I like to, you know, utilize that and implement them when I can and give shout outs, especially when they do such great work. And recently, I've been using, I've been creating file detections around things like lull bins, so living off the land binaries kind of stemmed from seeing a lot of attack chains where threat actors would utilize, bring your own vulnerable driver to bypass security features or elevate privileges and while host-based detections exist in perform well in those cases, before they're brought to your device, they do travel over the internet. So I thought, why can't we do the same thing? And so this is where I'm looking into specifically for drivers and executables. Most any legitimate binary will have a file version info. And that file version info data structure that provides the detailed version info and like company and description product versions stuff like that. So what you find, you find that at the pretty much at the end of an executable in the version info block, like I said, they would have a file version or a product version, product name, company name, some version of a description or even legal copyright. It doesn't have to have all, but it might have some of those things. And so what I do is I write a rule that looks for some of those values, some of those unique values. And more specifically, if you have like a version that is known to be vulnerable, known to be used in these scenarios, then I match that with product names or other unique values and create a rule based on that. So you can kind of take these characteristics that would apply to a lot of different types of executables, but not necessarily just the single sample or the single type of malware that you're looking at. You can use these shared characteristics to catch more fish, I guess, like cast the wider net to kind of take up some of that stuff. So it doesn't necessarily like, so for example, I'm curious, like this technique by leveraging file and poke characteristics to try and detect known bad things on specific executables. Have you ever noticed that that will catch things that aren't necessarily detected as malicious otherwise? Yes, yes, because actually this is actually one of those things where I did a little bit of work into it was talked about a bit before, but it's not a new or novel practice, but revoked code signing certificates. You know, we saw those with screen connect. We've seen those with other RMS that comes across as valid to some EDRs. They'll check just if the binary is signed, but not whether it's been revoked. And so we will, and I've written a couple signatures around that, which match on the signer and the serial number because both of them are not encrypted. They're plain text and working within the confines of Saracada and what it connects in inspect. It's possible to to notify when you're downloading something that potentially has a revoked sign certificate. You're like a cop on your over checking your ID and saying it's expired. It's okay. Go ahead. Tim, I love all of your analogies on this podcast. It's really like putting into context some of the work that Stuart does, which I think is often thought to be very technical, but there are some real world analogies that we can use to explain these. I like your cop pulling you over checking your ID and letting you go even if it's a fire. It comes from years of talking to people that have no idea what you're talking about. When you work in a sock job, you want to talk to friends and family about what you do. They're like, you're dead blankly. Yeah, you got to find out analogies. I'm sure Stuart does that all the time. Talking about your work at home. My nephew called me an internet cop when I try to explain, try to explain my job. That's hilarious. That's what my kids called with my job is all the time. They're like, what do you do? Tell them what I do. They're like, oh, so an internet cop. I mean, I can't arrest anybody. So, my cop, yes, that's a very common technology smell. That's super funny. So that's really interesting. So, are you able to, like, are there any sort of common executable types or different malware that is frequently using the same, either, you know, the file information, the sorts or things like that, that is there like a family that tends to use it more or a type of malware that you're having a lot of luck with this detection method? I've not been able to attribute to any particular malware or threat actors. I rely heavily on the other members of the proof-point threat research team who do a phenomenal job at tracking and campaigning and doing a lot of legwork just so that I can write the rules that I do write. So, actually, you explained your job so funny where we actually started recording this podcast. And I think you should share that with the audience because it's a great way of, you know, we've had a wide variety of people on the podcast and, you know, one of the people that we've had before, multiple times, is Greg Lesner, which of course, he's a, you know, an espionage threat researcher tracker guy. And I know that you guys were kind of, he's kind of a big deal. Yeah. And I know that you were talking back and forth and trying, like, you know, boiling down to the basics. Like what? Yeah, he, he called himself a professional data miner. And I was like, that's great. It's great to meet you. I'm a professional pattern matcher. And I feel like that, that really speaks to me on a lot of level. I honestly, I find it so interesting to find patterns. It's like being a detective. And like you said about fingerprints, and it's just, you know, every day, it's one of those things where you're like, all right, what's the mystery of today? What, what do I get to find? What do I get to use to, to, unlock the secrets of, you know, malware or what this weird, suspicious thing is doing. And how can I match on it? How can I, how can I be its pattern match? Well, and I think it's really interesting to you to think about the different roles when we're talking about cyber threat intelligence and threat research, because to your point, you're not looking at the attribution. You're looking at literally the bites in the code. You're looking at p-cap. And like, you're looking at traffic. And from our perspective, and well, and 10 is a deep and different researcher than I am. Like, we all have kind of our own sort of specialty skills. And, but yeah, I would also consider myself kind of like a data minor, but I also like doing sort of like overtime strategic analysis. So like looking for patterns in data that are like more in terms of like campaigns and malware as an attribution, and like looking at patterns and data of that type of thing. And taking like an overtime strategic analysis look at things while we have folks like Greg, who are like mining just like big type, like a lot of different types of data, finding new malware, finding new exploits, kind of, you know, doing a little bit, doing that type of thing, which then of course goes to you. But then I use your signatures to unearth more data that then I can collect and look at patterns in malware and attribution. So it's kind of like this like beautiful. It's a beautiful cycle. It's like the cycle of CTI. So yeah, I think I seen the diagram somewhere. Well, I didn't do a very good job of explaining it. There's more to and speaking of the CTI life cycle, there is one critical piece of the CTI life cycle, which never, ever, ever, ever gets fulfilled. And that's the feedback portion of the cycle. So we do hear within our own organization, but like I feel like oftentimes you don't, we're very lucky here at Proofpoint because we do have like a great, you know, feedback loop and really good like overall intelligence cycle. You can go feedback. I will plug for ET. If you go to community. Emerging threats.net, you can give the emerging threats team feedback on all these amazing rules that you may be implementing in your environment. And we love feedback because we want to make it, we want to make them better. We want to make them accurate. We want to make them perform well. We want to geek out on network traffic. Send us your pcaps. And let's talk. Yeah, you guys really do. It's great. I love the emerging threats community. I love how collaborative it is. We have tons and tons of external researchers, given you stuff. I think we've even had incidents where someone is like, Oh, I found this, you know, this new thing. It's not public, but I'd like to make sure that we have emerging threats signatures for it before it's released. Like out in the world, which is, which is also awesome. So please by all means send us send us your pcaps. Send us your data. Not us as in me. I won't be looking. But I mean, they could send it to you, but they can send it to us. Yes. Yeah. Yeah. I'll be the middle man. I will, I'll be a, I will happily be the peak, the peak at Mule. Peacat Mule. And we also like feedback on this podcast. So if you'd like to comment, go, go find us on iTunes or, or on our website. Whatever, give us a feedback. We'd love it. We actually have. We've recently added a text us module. So if you look at the description on wherever you're listening to your podcast, you can text us your feedback and we will happily incorporate it into future podcasts. And this was super interesting. Thank you so much to our, we will definitely have to have you back because we had a list of things to talk to you about. And there was a whole section that we didn't get to, which was new techniques in different SDG smuggling and SMTP detections, which is also super interesting. So we will definitely have to have you back for any of our listeners who are interested in learning more about emerging threats, learning more about how to write your own signatures. Definitely check out community dot emerging threats. There are tutorials, tips, tricks, you know, you can message the team directly on, on various platforms. So definitely reach out. And one of the greatest parts of the emergency best team is it is education for the community. So so Stuart, thank you so much for joining us today. This was a really fun conversation. It was super interesting. I learn a lot every time we have an IDS network is sig writer come on the podcast. So it's very exciting and I will send you things if I see commonalities and file impose for different executables. That's yeah, that sounds like a really cool little detection trick. So good to learn. All is always thank you so much for being the co host today. We're going to good vibes and metaphors analogies and speak easy. That's what that's on here for ice easy. Yes, yes, amazing. And 12 listeners as always, thank you so much for tuning in. And until next time, happy hunting. You've been listening to discarded tales from the threat research trenches, a podcast by proof point. Never miss an episode by subscribing to the show in your favorite podcast player. Happy hunting!