Post-Breach Action Plan: When Systems Go Down for Days w/ Drew Simonis | Saket @ CISO Confidential

you know, the notion that a security breach always happens at the worst time is not always true. Not only do you learn that they're worse than you thought they were, they become worse than they started because we want to make sure we focus on the disaster and the cleanup and we don't want to inject too many distractions. So what happens if the same business unit leader comes and says, hey, you think my system down? Yeah. Do you also tell them that no one right now is not the right time? We would love to jump in and talk to us about a hack which you remember, which you profoundly or not super fondly remember as in your career of a few decades now as a security leader. We'd love to start from there. A security event that we thought was fairly mundane. You're looking at potential scanning activity, probing type reconnaissance stuff and we were able to detect that and this happened in a few days before this was this was before the idea you're talking about before D day, I got it. And so, you know, we pick up this reconnaissance activity, we pick up the scanning activity. When you see a pick of this happened in your sock and yeah, our sock was, they got it. And they didn't actually get to me right away. Okay. Because they were able to what they thought was interdict the activity. So they saw scanning, they saw the probing, some of you scanning the porch and yeah, that same IP addresses, they did some counter scanning and they said, you know what, there's some problems on this box. Okay. Looks like maybe those two things are going to intersect. We need to take some action. And as a business, usually they're as usual, yeah. And so they went through that process. They took an action. It turns out that the action they took was, I would say ineffective, but it wasn't comprehensive. Okay. And so the attacker had the ability to take further steps to sort of move up the ladders. What we see a lot is attackers take the easiest path that they can take. Sure. They're they're they're I know. Yeah, exactly. But the assumption that that's the only path is a dangerous one. Like they have the ability to elevate their game if they need to, if they're targeting that system. And a lot of times you're not facing a targeted attacker. And that's what causes that perception that well, really all they have are the lowest hanging fruit to pursue because they're opportunistic and they're just trying to, you know, whatever door I can open, I'm going to open and I'm going to see what's inside. In this case, the attacker was more targeted and they were looking for specific activity and specific things. And so they were able to step up their attack and and so a system that we thought we had brought around to health, you know, prompted by the activity of the attacker to investigate in the first place was then several days later compromised more fully. And and so why are you monitoring it? Right, right. So we had eyes on it, which is good because if they hadn't started without probing type behavior, we might not have been paying attention. But but that that I'll call it secondary attack was far more effective. They were able to get into the device into the box. They were able to start probing the data and and at that point, we had to take much more aggressive action. Okay. And and bring down a critical business system. Oh, okay. Without consulting with the business. Well, you know, and so you've got to have this is where your your tabletop exercises, where your prep work, where your policy all come into play because you have to have an and my instant response lead knew she had the the latitude to make that decision. Didn't even insult me. Didn't have to consult me because this was something that was going to get far worse. Okay. If we didn't take an immediate, let's take a step back. If you don't mind, when did you find out for the first time because your team was handing this? Yeah. When did you find out for the first time? What are the courses of actions that you took? One is communication to your leadership, internally communicating, externally communicating if you did. Just walk us through that day. How did they unfold? Yeah. So so once there's a little bit of the fog of war clearing and people can understand, like, I got breathing room like, okay, we took a drastic action. But that drastic action has cut the attacker off. Now, now there's time to notify me. Okay. So you didn't even know like the D date. This is D zero minus one. Yeah. It's two days when this is well, it's I knew the same day. I knew the same hour. Okay. But I didn't have to approve the action. Okay. Got it. Right. So my team is empowered to take action to stop the attack in a tracks to the best of our ability. Okay. And this is a large company, a large company. A large company works for a large company. Yeah. So it's one of the large companies. Yeah. And you know, it's one of the business units. So stopping a critical business unit or maybe a system there without consulting you can be a very big decision. It is big decision. So it's it's it's necessary. You know, attackers move at a pace where if we hadn't taken that action as a team, the consequences would have been measured in the millions of hours. And and and so, you know, you want your people to feel paralyzed. I've got to go eat your approval. Each second that's ticking away, money is going out the door potential for catastrophic loss. Kind of. Or do I want as a leader who have to go and explain something to one of my peer leaders? I think I'd rather be in that situation explaining, you know what, even if it was a mistake, maybe we took your system down improperly. We shouldn't have done that. We're going to learn from that. I'm happy to have that conversation. Rather than we didn't take your system down because we wanted to be cautious, but but we have to, you know, pay a million dollar ransom or whatever that might be. And and so, but then then we have to start dealing with the cleanup. We can't do those kind of things without engaging with the business eventually. And so as I get notified, then we have a whole other structure that comes into play to support the event. And one of the incident response plan and business continuity plan is so important and crisis management and crisis management. The other dimension is also setting expectations. Yeah, but this is where risk quantification comes into the picture. Just where, you know, a lot of people don't really understand if something goes down, what are the costs which can be involved? Yeah, is it a 10 million impact or a 100 million impact or a billion dollar impact? Because we saw United Health, which is now over two billion dollars of impact from one hack just to United Health. And if you take everything else, it's obviously much more. How do you set expectations in terms of, you know, what's the likelihood and what is a potential impact of an incident? Have you seen that happen? How do you look at like setting expectations with, especially your leadership, business unit owners, especially when somebody's trying a new AI tool, not do you look at setting expectations and getting them on the same page when it comes to your cyber risk? If this, like if this bad actor doing this bad thing was able to do this against you, what would that feel like? You know, it's not always for me fully quantitative at this point, because the maturity is not there necessarily. But we can start talking, I'm a big fair fan, and I'm a fair fan, even if you don't use fair, because I think the way that it allows you to decompose an issue into those component parts that are very easy to rationalize with the business. I think that alone, the autology, is a very powerful part of fair. And so being able to sit with a business person and say, right, we know there's a pretty good chance that this bad chain of events is going to be successful against your critical business system. And we know that that feels bad. Let's talk about the components of that bad feeling. Like you're going to have regulatory penalties, or you're going to have customer breaches, or you're going to have a cleanup from that. Do you have a real effective business continuity plan, or are you going to have to do a unnatural act that are going to be expensive and time consuming? And that allows them to think about their system in the context of this challenge in a way that is for many of them, the first time they've had that kind of deep thought, and it's eye opening, it also builds a bridge between us and them. Because we're not talking about cyber necessarily. We're talking about cyber as a catalyst, but we're talking about them and their systems in their business. And so they're very comfortable having that conversation. Like they might have to think deeply, but they can do that. They know the answers if prompted and given the framework could be prompted within. And so then once you're good at that and start quantifying that, if you do face a regulatory penalty, for example, what's that going to look like? If you do run the risk of losing customers, which customers, how important are they? How strong is that relationship between you and them? And can you recover that? And what cost do you recover that? So part of that preparation for that incident is the journey towards a more robust risk management framework using something like a quantitative approach. Exactly. And that helps you prioritize. Yeah. And you can actually then do these so-what analysis out. What happens if I invest in a backup? What happens if I invest into EDR and then actually see the ROI and then take it to your CFO, your CEO, your board? Yeah. And make their decision to say, if you give me this much money, this is the amount of risk I can burn down. Yeah. We'll be better. Don't give me this much money. We're living this fine. And I'm okay. But the job of a CFO is to show the risk. They don't own the risk. That's right. Owners are the CFOs and the business owners because it's finally their decision. Yeah. And that's it. A lot of people don't get that. You see that in your peer group? I do. I think I use the analogy of a risk magnet. I think that there's a lot of people who are in the CSOC who want to grow their organization. Yeah. Politically, there makes a lot of sense to that approach. Sure. The bigger you are, the bigger you're even at orgies. Yeah. And so they approach the role as, and I'm going to take your risk and make it mine. Yeah. And that's great too. For a business person who's not savvy, they're happy to give it to you. Not happy to give you resources to execute on that mission, but that's your own problem to solve because you just told me that you can do this for me. So now I've made my problem, your problem, and I'm going to immediately turn around so our old and you accountable for delivering a solution that you probably can't do. And so you undermine your own credibility as a leader. Like anybody, if you say, I'm going to wash a car for you, go for it, you know? If you have soap and water, that's great. If you don't, I expect a clean car anyway. And so we do ourselves a disservice by this behavior when we really should be educating people about the risk they're taking and helping them take those risks responsibly. And guess what, every car owner wants a clean car. Yeah. And everybody wants a risk down. Yeah, everything is the better. Yeah. Right. Like they don't want to be in this event scenario. We talked about earlier where they're, where they're uncertain about their ability to do their job early. And and that's like I say this all the time. The role of security is not securing things necessarily. The role of security is making sure tomorrow looks like today. Wow, people just don't want to wake up to a surprise. We don't want to do it in the event context. Business wants to, you know what, I want to be able to focus on my own strategies, my own people, their development, and my own goals. Like I don't want to have to suddenly be in the midst of your world because somebody screwed up. And that's by the way, more of a reason because when you tell what is the risk and risk is always predictive, you're done with the probability and what can be deemed back. Yeah. You don't surprise. Hey, I was expecting the $3 million impact while in reality, it was $30 million. That's a surprise. That's a bad one, right? And that's a surprise, right? Unfortunately, a lot of them happened that way because you were saying it was a high risk. Your definition of high and my definition of high, like show yeah, it's even the night. Exactly. Yeah. Fantastic. This has been incredible. I will wrap up by asking you three pieces of advice that you would give to a young security, you know, up and come or that you would say that aspires to be a senior seasoned industry leader, see so like your Sarah. What would those three advice as beef? It's the most important and maybe the most controversial. I've even folded my arms for this one is don't care about security as much as you think you need to. Wow. Okay. I think that getting too close to the problem robs you of empathy. And so your ability to communicate with your business colleagues amount security is limited, like you get very myopic. Wow. And so if you can't step away and see the problem in the context of all the other problems. And that's what I mean, it's not that we don't love what we do. Don't try to make security the preeminent problem amongst all the other business problems because revenue, new product, introduction, these are all also very important problems. And you have to see ours in that context. And that empathy is super important. And so that's probably the most important piece of advice. The I would say the technology is always going to change. People are always what are the same. And so learn about the problem from a people perspective. And while you need to master technology or recognize that there's plenty of things you can do without any technology. And so learn sales skills, learn influence skills. Because as a leader, your real job is to is to advocate for your program. And then the third piece of advice is tied to that thematically, don't ever go into leadership if you're not willing to advocate for your people. And focus on their development. Not seeing your people as a tool for your own propulsion forward, seeing you as the way to propel them. And so if you don't have a real keen interest in developing your team and helping them reach their goals through the organization, the priorities that you take, then you'll be a selfish leader. And the selfish leader will never be empathetic. You'll never build the relationships. You'll never be effective. And so you have to look at yourself and go, you know, I love the technology. Find a place where you can be a technologist. And if you say, I love the people and I want to make them the best, I think you'll be a successful leader. And Tasty, this is such incredible insights. Thank you so much for being in honor to be linked to you on this show. I've enjoyed and I hope you had fun. I did. A kid, thank you.