Go back

Grid cybersecurity: how big is the threat & what should we do about it?

72m 23s

Grid cybersecurity: how big is the threat & what should we do about it?

The conversation between host David Roberts and cybersecurity expert Patrick Miller explores the real and exaggerated threats to the US electricity grid. Miller clarifies that while there is documented evidence of hackers breaking into US grid systems, they have not yet caused direct damage like blackouts; such attacks have occurred in Ukraine and Poland, perpetrated by Russian state actors. A core theme is the distinction between IT and OT systems, where OT devices—the physical interface to grid operations—are uniquely vulnerable because they were not designed for connectivity and cannot easily be rebooted without risking outages. The NERC CIP standards, which Miller helped create, mandate rigorous protections for the bulk power system, including firewalls, physical security, and supply chain checks, enforced with substantial fines. However, these standards do not cover the rapidly expanding distribution grid, which includes distributed energy resources like rooftop solar and virtual power plants. This creates a fragmented regulatory landscape across states, where the attack surface is exponentially larger. Recent accidental disturbances from inverter-based resources in Texas and California have heightened concerns about potential malicious exploitation, leading NERC to study these risks. Miller emphasizes the need for unique isolation strategies and proactive management to address these emerging vulnerabilities, balancing between hype and complacency.

Transcription

13102 Words, 72623 Characters

English
[Music] All right, all right, hello everyone greetings, salutations, this is Vultz for August 7th, 2026. Grid Cybersecurity. How big is the threat and what should we do about it? I'm your host David Roberts. As clean electrification proceeds, more and more of the US economy is going to be dependent on the electricity grid. And the electricity grid is going to be more and more dependent on power electronics, software and computing. If there's one thing we all know about computers these days it is that they can be hacked, accessed and controlled from the outside by bad actors. Thears about the vulnerability of the electricity grid to hacking and manipulation have been steadily rising in recent years. Most of the power electronics, the inverters, transformers, controllers, used to build the electricity grid come from China. If you listen to the Doom sayers they'll tell you that China is embedding malware on all these devices with the goal of creating some sort of apocalyptic kill switch that could take the entire US grid out at a stroke. How much of that cybersecurity threat is real and how much is hype? I've been meaning to get to this subject for years and I'm excited that I have the perfect guest with me today. Patrick Miller helped write the original cybersecurity rules for the US bulk power grid, the NERC critical infrastructure protection standards. In the 2000s, then he became the first person in the country with delegated federal authority to enforce them, auditing utilities on the government's behalf. Since then he has founded a nonprofit, run a Department of Energy program and built a consulting firm all to advise and educate utilities and regulators worldwide on cybersecurity threats. It's an unusually broad range of experience and it has made him a prized voice in the field, a level head in an area filled with uncertainty and fear. I'm eager to talk with him today about how cybersecurity threats might manifest in our increasingly distributed grid and how grid engineers and regulators should be managing those threats. All right then, with no further ado, Patrick Miller, welcome to Voltz. Thank you so much for coming. You bet David, thanks so much for having me on the show. Really excited for this. Something I've been circling around for a long time, as I said, and I'm excited to dig in. So among other things on your extensive resume, you run a program for NERUK, the National Association of Regulatory Utility Commissioners, a training that brings state utility commissioners up to speed on cybersecurity. I guess where I'd want to start is, when those commissioners come in, I suspect that they, like me, like I suspect most of my audience, don't know much and have all kinds of weird preconceptions. So I'm curious, sort of like, what's the first thing you kind of tell them to get grounded? And what sort of misconceptions are they carrying around with them? Out. Great. Plays to start. Yeah, I do a lot of work with NERUK to help the state commissioners. As you know, where the federal regulations don't reach, it's pretty much left to the states. And in a lot of cases, you can end up with like, you know, 50 different directions. Yeah, we're going to get into that later. Okay. So typically where I start with them is they come from varied backgrounds. They're almost always a legal background. Or in some cases, they may have some infrastructure. They may have been a, I've seen CFOs and other, like even general counsel from various utilities, whether it's telecom, water, gas, electric. You start with the fact that they probably don't understand all the technical things. And the typical questions they come with, they, they usually sit on one side or the other of a pretty, you know, wide divide. They either think that, you know, everything's fine. Nothing is hackable. This is all a bunch of hype. Or they think, oh my god, everything is hackable and China is here to eat your children. You know, it's just, you know, so it's kind of one or the other. There's a few that are in the middle, but it's, it's a swim few. That's funny. That's funny. Well, hopefully we can find some chart, some course in between, in between those. That's my goal. Yeah, yeah. So before we get into some of the details, maybe just a level set, I kind of want to know, are we talking about something that might happen or something that has happened? Like have there been notable cybersecurity attacks on grids that have had negative effects? Or is this mostly something that people are worried could happen? Well, there's a bit of nuance in the answer. There have been attacks on grid companies. We do know that there's public evidence of that. Most of these have been situations where the attacker is broken into this system and hasn't done any damage yet. They just preposition themselves. They're holding that access. So in theory, they would use it for something, you know, bad or nefarious later on. What we don't have is direct examples in the US where, you know, actors have broken in and actually directly caused damage. We do have evidence of this in Ukraine more than once. So at least a couple of exercises there. And then one in Poland recently, and when I talked to my contacts in Poland, they say they've got a long list of things they just haven't published yet. So there are some threat actors that are definitely breaking into power systems and definitely causing direct damage, direct harm, blackouts, that kind of thing, just not in the US. Just curious in Ukraine, is that, are those state actors? Is that Russia doing that or are doing no? Yeah. No, it's definitely Russia. Yeah. Same with Poland. So this is something that's being used as a proxy for warfare kind of thing. It is. It's a proxy for warfare. It's also proxy for influence. So, you know, Poland is the major channel that Ukraine gets supplied through. So Poland's also been under heavy attack from Russia as well with basically the same system, same type of attacks. Okay. So for the US though, we've documented access, but we've not yet had anyone like cause a blackout, say or something like that. Right. Right. Okay. So I thought I'd start with a basic distinction here. I think a lot of people, when they hear the word cybersecurity, they think about IT. They think about information technology, protecting passwords, protecting data, preventing identity theft, that kind of thing. But a key distinction here is that, is that cybersecurity in the way that you are dealing with it is mostly about OT. I am operational technology, physical processes, and those are different, the different approach. So talk a little bit about that distinction and how it's meaningful for cybersecurity. Sure. I mean, IT still plays a role. I mean, information technology is how any company runs these days. You pretty much can't work without your email and your corporate systems and your data. You know, all your customers are in there. All of your billing is in there. In the OT world and the operational technology world, it's the actual interface to the physical world. So like the sensors that take things like fan speed or oil viscosity or temperature. So this is like, you know, think of how much voltage is on the wire, for example, that's sensed by a digital component now. And the OT is what directly touches the physical world. Like it's the last line. So you would click on a screen somewhere, like a picture of a breaker. You click on it and it will open. An actual breaker out in the field. And that travels from someone's IT system, like a standard workstation. It's a Windows machine with a program on it that runs kind of an operational window into what they've got in their field. So they've got transformers and breakers and all the things they use to operate their power system. They can operate those from like a control center through this kind of IT OT partnership versus send someone out in the field to actually go open that breaker. Right. So that's the information that we get with this OT. But these systems are not like the OT systems. They're not like, you know, Windows or Unix or a database or a web server. They're literally purpose built devices. They do one thing. They like they open the breaker. They sense the voltage. They don't have like an operating system. They're not, you know, they're basically just what's called firmware and a configuration file. You just turn it on and it reads a configuration and it does its thing. So they're very interesting and it seems like it seems intuitively like that would be a little simpler and maybe a little easier to protect. Is that right or wrong? It's easier to hack as well. So because there's not a lot of, I mean, they weren't really built. I mean, think about this. When we first built this out, they were never really connected to anything, right? They were just by themselves. So now that we've networked everything and we've made control centers that can control, you know, everything in the field. So, you know, generation assets, transmission assets, distribution assets, with all of that connectivity, you end up with devices that were never really meant to be connected that have some kind of connection to them. So they weren't designed to be protected in the same way. So we have to come up with really unique. and interesting ways to isolate them and limit access and that kind of thing, because realistically, to update them, you have to reboot them in almost all cases. If this thing is required for the grid to run, you don't just reboot it, because you get a blackout. So you end up with, you got to wait for the right moment. So you have this kind of mishmash of very distinct. So our main protection is just to isolate the heck out of them and architect it so that it's very difficult to get to them unless you know what you're doing. Okay, talk a little bit about these. We're going to start with the bulk transmission grid. We'll get to the distribution stuff later, which is really my true interest. But let's talk a little bit about the NERC CIP standards, the critical infrastructure protection standards. So you were instrumental in developing these and then enforcing them. They're about the bulk transmission grid. Just give us a little texture of like, what are they involved? What do they require of people? And they're sort of unusual, I mean, the grid world, the electricity world, full of all kinds of standards that are mostly voluntary. These are not voluntary. They are mandatory and enforceable. So talk a little bit about what they consist in, what do they ask people to do, and how they are enforced. Sure. Let's start with who is required to follow the law? Because everyone thinks, oh, it's the whole US power grid. There is no distribution in scope at all. And it's only some generation and some transmission. So there's a handful of exclusions for transmission. So it covers a pretty big area of the transmission system. Generation, if it breaks a certain threshold, which to give out some numbers, it's 75 MVA aggregate behind a single point of interconnection. They have to register with NERC and they're required to follow the laws. There's a lot of generators out there that aren't 75 MVA. So in a lot of times, they would even generate facilities to be 74.5. So they could get right under the threshold. But that's changing. We're taking that threshold down. It's going to go down to 20 MVA and connected to actual distribution environment. So the thresholds get lower because a lot of those smaller generators have actually had an impact. So that's who's got to participate. The rules they have to follow, basically, they have to declare which systems are critical. And these are systems that if for some reason, there was a problem whether you needed it or it got hacked. If it would cause an impact to that system within 15 minutes, that's a critical system. You got to protect those. And protecting them means you got to have security policies in place. You got to background screen your people and train your people. You got to restrict who has access to those systems and you got to remove their access when they leave. You have to put them inside of what's called an electronic perimeter. It's a firewall, really. And most people understand what a firewall is. It's kind of a way to protect the systems from other dangerous networks. Then you have to have a physical boundary. So you got to lock the systems up. You can't just let anybody have access and track who has access to them and go on, I'll see if I can get my hands on it. I can hack it. Then you do the systems themselves. So you secure each individual system with everything from like password security to anti-malware, to logging and tracking and monitoring. And then of course, you have to have incident response plans. You have to test your plans. You have to have backup plans in case you need to restore a system and test those. And then you got to restrict their configuration. So if you want to make a change to the system, you got to track them all, prove them all. And then there's the information about the systems that if hackers got it, would be really bad. You have to protect that information. And then there's supply chain security. So you don't want to buy from North Korea or China. Or say you got to like, well, do we really want to buy from these people? Or we're taking a big risk here. The next piece is the communication between control centers. There was a Chinese hack recently on-- they basically hacked all the different telecoms. So control center to control center, you have to encrypt that. So if someone were to somehow get in the middle of that communication, they couldn't see what's happening. And then the last piece is what-- there's some physical security additional stuff for like really, really big-- sort of transmission substations. The last one that just got adopted is monitoring the traffic between those critical systems as they talk to each other. So we can see what they're saying to each other. And that's the last piece. That pretty much covers what the requirements are for the body of the standards. That seems like a pretty-- that's a lot. That seems to go-- It is a lot. So these are mostly large entities being asked to do this, large power companies, transmission-- Even some smaller ones. I mean, if you have transmission, like if you break 100 KV, and you've got transmission, there's a chance you're in scope. And if you're a generator above 75, then you are in scope. So it actually covers a fair number of the utilities. It's just which systems fall into what they call high medium and low, because there's some categorization in-sip. Most of the systems are in the low impact space. So they don't have to go through all of those things. They've got to go through some of those things. But if they're medium or high, they've got to do all that. And it is a little bit of a subjective question. But how big of a pain in the air is that? I mean, if they're building systems that are 74.5 KV, this probably indicates that they'd rather not take all this on. Is this something that you can hire a guy who will do this for you? Or is-- I mean, how elaborate of a process is this? How much time does it take for them to do this? It's pretty elaborate. And what they really expect is they really expect kind of a level of diligence. And the reality is, if you want to participate in the grid critical infrastructure ride, you have to be this tall to ride the ride. And the expectation is high for a reason. And when I was auditing it and writing violations and enforcing, we took it very seriously in the fact that, no, this is the North American power system. You have to keep the lights on. So yeah, you've got to have a real program with real people doing real things. And you've got to take it seriously. And presumably, these fines are big enough to really-- these are not just things that they could absorb and brush off. The number that gets thrown around is a million dollars per day per violation. That's actually gone up with inflation. It's over 1.8 million per day per violation. That's never been seen, though. So what they typically do is they'll write the penalty. And if the utility is like, oh, OK, what if we put all this stuff in place and we fix these things, we put this money toward improvements? Can we take that cost down? Usually, that's how the settlements work out. So that it's less of a go spend a bunch of money on the penalty and then go spend a bunch of money on fixing it. They try to get them to incentivize the fixes instead. So it works out in a pretty good way. It drives the incentive to do better. And who is this we that is doing the enforcing? Is this the federal government? Is it some department of the federal government? Who's the enforcement arm? It's a branch. So the authority comes out of FERC, the federal and the Regulatory Commission. They have basically delegated this down to NERC, which is the North American Electrical Liability Corporation. They're now called the ERO or electrical liability organization. So they do the enforcement stuff for FERC. And there are six regions of NERC. So whatever region you're in, your region audits you. And if you're in a bunch of regions or all regions, there are some utilities that are big enough to actually cover all regions. You get audited. They kind of take turns and one of them will lead, but the rest of them participate. So yeah, you get audited by your region and then that goes upstream to NERC and to FERC. And then the penalties come back down. Okay, interesting. Okay, so here's the bit I'm really fascinated by, which is the CIP standards were designed for the bulk power system. But as listeners of this podcast are very aware, all the action these days is out on distribution systems, all the stuff that we cover every day here, rooftop solar batteries, EV chargers, virtual power plants, sit outside that sort of regulatory perimeter. And in states, as you say, beyond FERC jurisdiction. And so you get something like a virtual power plant, you get the size and scale of a power plant that might have come under the bulk standards, but instead of a single point of regulation or access or physical, whatever, the VPP consists of thousands, thousands of devices scattered all over thether and Yon, which just like intuitively to me sounds like a cybersecurity nightmare. (laughing) You have your attack surface as they say, has become exponentially larger. So I'm just curious like what are grid, what's the cybersecurity community thinking about this? How do you reach these distribution grids? What sort of standards do they need? And who's gonna enforce them? Like how do you standardize an approach when you have 50 different regulatory bodies involved? - Yeah, this one has been a really interesting challenge. So you've got some examples where those smaller assets, whether they're run by VPPs or in some markets, like in Texas where it's kind of a QSE market, and what we now have just lumped them into a big term called inverter-based resources, because it kind of captures all of them under this umbrella term. So there have been inverter-based resource, we'll say challenges, disturbances, also known as blackouts, in Texas a few times as a result of these. So they're one of the good examples. There's been some other disturbances and things in the California area. It's called, some strain and some other ones. We've got a pretty good-- - Are we talking about accidents? Or are we talking about attacks? - No, no, these are just like grid physics problems. - Right, right, right, right. - Yeah, so not attacks, let's be clear. So what they did just because those grid physics problems manifested, they were like, well, what if someone caused this to happen with hack? That would be even worse. So those two things converged and basically, NERK went out and did lots of studying and invited lots of different participants and stakeholders to the mix. And they came back with this inverter-based resource study. And as part of that, they took that threshold that we were talking about, that 75 of MVA for generators down to 20 MVA. So, and it used to be 75 MVA connected to a 100 KV system, which is basically transmission. They took that down and now it's 20 MVA connected to a 60 KV system, which is distribution. - So give us a system, what is 20? Like, would that include like a residential rooftop solar generator? How big is that? - It wouldn't include, this is like a small scale, like if a whole, a small neighborhood, for example, or a, it's mostly commercial operations. It's not likely going to be rooftop. It was mostly generated for people that maybe had some battery storage that they were feeding back in, for example, they were on the market as a result of that. So it was trying to capture those that are big enough to matter and they looked at all the way down to rooftop, like where did we draw the line? So they took it up to 20 MVA because they figured this is big enough in aggregate if someone were able to control it, it could cause problems. So based on both the grid physics and the security aspects, they drew this line for what's called inverter-based resources. They now have to register, they have to do some things, they don't have to do security things yet, but that's likely going to be coming in the future. But we're starting down the path of roping them into those federal regulations so that there's less of a state by state mix because as you've probably covered, various states have their own rules about how things are done. Yeah, how does that work? I mean, how does the, how do the feds even have jurisdiction here? Doesn't, isn't that violating sort of the, that kind of state federal boundary that everybody cares about so much in this, in the grid? Yeah, this has been, it's been hotly contested. I mean, it's, it's, I mean, nothing short of a bar fight, really. But what, what the, the result of this was, and Ferric has pretty much put their foot down and agreed with it, is that in these situations, it's affecting an interstate transmission system as a result of their ability to influence it with those smaller resources. Yeah. So the aggregate possibility has, they've, and they've actually done a pretty good job of proving it with physics. It's not just a speculation, they can, they can show their math. Yeah, this is, that, that line has become more arbitrary every day, just, just around the physics, just around the technology of everything. And I've been wondering, I'm sure this is a species of problem that is going to pop up more and more that line between what is and isn't interstate, what is and isn't bulk is, is fuzzy, at best. Yeah. Yeah. And we haven't even talked about like VPPs that could control assets all over the country. Yeah, right. I mean, a VPP that's aggregating resources across multiple states. Yeah. So, you say right now it's about physics. What would, I mean, I guess this for the nightmare scenario here is like, you gain, you know, a hacker or whoever, you know, and I, again, speaking here from a very low base of knowledge, but like a hacker gets access to one battery through an inverter and like via that creeps into the other batteries it's connected to and creeps its way, you know what I mean? Like it gets in a back door through one thing and then accesses the whole system through that thing, which makes me think then that like you have to ensure that every one of those batteries has protections. Like is that the threat here? Yes. That's a bit short and sweet about it. Yeah. Well, we did some work with Nairu, because we mentioned earlier, we did what's called the cybersecurity baselines for distribution and DER aggregators. Because those DER aggregators are a lot like a VPP, in a lot of cases they aggregate at a point much like a VPP does. So when you've got virtual power plants and distributed energy resource aggregation points, you have at your fingertips a very large amount of potential to affect, you know, regional problems, just because you could say, well, you know, if you did it across the whole board, yes, that would cause problems. If you want to focus in a certain area and cause a problem at the right time to cascade things, you could do that. So we wrote some standards and it was basically some very lightweight things, you know, like have passwords, don't connect these things to the internet directly without a firewall, really simple stuff, kind of basic things. Like, you know, if you're going to have remote access, put some multifactor. Give them a one time, you know, phone code to actually log in so that you can't just like log in with default credentials. Are there any of the, 'cause one of the things that pricked my ear about the bulk, about the book centers, is that I hadn't really occurred to me, 'cause you know, I was thinking about kind of the technology angle of it, but the personnel angle is also obviously a thing. Are there any of those personnel based standards going to be applied down at the VPP level, the aggregator level? So far it hasn't gotten to that stage. And states can certainly go do this on their own, right? And that's kind of what it was designed for. When we wrote these, what we call the baselines, we based them on a DHS work called the Cyber Performance Goals. And it was just lightly tailored to fit the distribution environment and for DER aggregators, but it was designed so that states could use it as a model to actually go right their own regulation. And it would give them something to work from a blank page as always the hardest place to start. Yeah, right. And the ones that want to go further and do, you know, go their own route and do their own thing, they're going to do that anyway. But there's a bunch of them that don't really have anything to work with, so that was designed to give them something and hopefully honestly get some degree uniformity in the approach. Yeah. So it's like a floor, a floor then, a federal floor that states can build on if they want. At this point, it's just a suggestion. It's a nice to have. Yeah. Interesting. And this is another just sort of intuition of mine, which is that once you're down to like 20, that 20 KV level, you know, you're not talking about individual residences, but you are talking about some pretty relatively small operations. Like if I'm a, you know, I could have a warehouse, you know, with a bunch of solar panels on my roof in a stack of batteries, I could break that threshold. Yes. So you're, I just feel like there's a limited amount. You're going to be able to demand from like a mom and pop. Like somebody who owns a warehouse, like these people are not going to be, you're never going to get them to be cybersecurity experts. You're never going to get them to hire dedicated personnel or employees just to focus on this. Like, I'm just wondering, like how far can you get? I don't eventually, you need to start building the safety into the devices themselves. Is that not sort of like where this needs to go eventually? Yes. And we're also trying to do that at the same time because the way the system is designed is you've already well articulated. There's not an easy way to do this. You can't just make all the warehouses out there with rooftop solar and, you know, warehouse level rooftop solar and batteries, which most of them kind of will break that threshold. You can't make them become like cyber secure and follow these federal guidelines and get audited, for example. I mean, there's not a lot of the audits, right? So, you know, on the reverse side of that. So what we're also trying to do is to actually, you know, do this at the supply chain level. So I've testified to Congress on this and some of my other counterparts have. Dr. Stewart is another really smart one, Emma Stewart. She's brilliant on this subject. Probably the global expert on this really. We've taken a strong push to get the device level security to a place where you don't have to be a cybersecurity expert. Right. And this is not an easy thing to do because we buy a lot of our equipment that, you know, not from the US. So imposing rules on China is a little challenging. Yes, yes. And also, I would imagine that like, for every barrier you put up to hackers or bad actors, you're making some incremental trouble just for the people who are just trying to use the device. You know what I mean? Like, one more little gateway for the normal everyday use of the device, which I'm sure like the people using the devices are going to resist somewhat and the manufacturers are going to resist somewhat. So this does seem like a really sticky wicket. Yeah, it's not an easy one. But, I mean, we have to try all fronts. And if we get an inch out of all the fronts, we actually can cover a larger humanitarian. So it just takes a whole village of effort to try to make the needle move really. Well, you've set me up for my next question, which is, as I think people probably know, most inverters, you know, we're talking about inverter-based resources. Most inverters come from China right now. There was this big Reuters investigation in 2025, which said that inspectors found all kinds of little undocumented communication devices inside solar inverters and batteries. And I guess like two questions. A, what do you do about what I imagine are now hundreds of thousands, if not millions, of those Chinese inverters that are already installed somewhere and running, like you can't yank them all out, you can't, there's no practical way to audit them all, what do you do about those? And then what do you do about the new ones? As you say, we have no jurisdiction over China. Like how do we exercise any control over what China puts in those inverters? - Yeah, I'm gonna keep this at a non-technical level 'cause it gets really technical really fast. So in general, you're absolutely right. There's millions of these things out there, and they do have a lot of undocumented communications components in them, but I wanna make sure we're really understanding what that means without any hype whatsoever. When China manufacturers a lot of this equipment, they literally do it on what's called a commodity board. So you can imagine in your mind like a motherboard, right? It's a computer component where the chips and the memory and all those things sit. And they'll buy literally millions of these at once. And they'll buy them from a commodity manufacturer that just makes boards, literally like generic board. And on that board, it'll have all the things just in case you need them, whether you use them or not. They don't buy specific boards with just these components. So when they, I mean, 99% of the ones I've seen that they've gone out and found stuff, it's just been the commodity board. Yes, it came with a radio, it came with a motor, it came with a place for a SIM card. There wasn't a SIM card inserted, but it has all those components on there in case the buyer of these commodity boards wants to use that functionality. I see, I see. So not nefarious inclusion of these, right? Not necessarily. So whenever you see those, you kind of have to take that with a grain of salt and think, well, if there's millions of these things out there, did they just use commodity gear? And they're just not, you know, this is just part of the board that maybe it's used, maybe it's not. And now they have found some of these things phoning home. And, you know, the routing traffic through China. So that's a different discussion. That's at the software level, you know, up at where the applications that you're using to manage these devices and that kind of stuff. So those are different components. And there's some supply chain there that we can impose because it's software, right? We can inspect code. We can enforce certain rules on how code is developed and what's, you know, certain, like, transparency of the code. We need to look at the code before we're willing to allow it. These kinds of things. And we're not just going to let updates from China happen without someone taking a look at it before it goes in, those kind of things. So there are some gates we can put in place. You say there might be these elements on the boards just because the boards are sort of generic and created for maybe lots of different purposes. Could those, could those components of the board that aren't being used be activated later? You know, I mean, it's sort of remotely activated. Like, that's, I guess, is the worry. Yeah, they're there and they could be activated. But they'd have to have, I mean, you could activate them but you'd also have to configure them. Like, it's hard to explain, but when you, just because you turn something on, does it mean it's ready to be used? Right. Like, even a phone, you can turn a phone on but you have to give it a SIM card. That's got an activation code. That's got, you know, all these things have to happen before it'll actually work on a cell network. I mean, you could get a phone out of the box and turn it on and it just doesn't do anything. So those, those pieces of gear are there that doesn't mean even if you did enable it that it would automatically work. Right. So it does, it ups to anti in terms of just like flipping a switch and getting this like rogue network of these things to go do stuff. Right. So are we currently trying to impose standards on these imported inverters? Like, is that something that's happened? Is gonna happen? Is in the works? Yeah, states like Texas and Florida have like banned certain Chinese gear, the feds of banned certain Chinese communications gear. What we're trying to do, at least myself and some of my friends that are pushing this, we're trying to get a place where we can actually just get transparency into what they're selling. It doesn't make sense to go rip and replace all this gear. They actually make quality stuff. I mean, we used to have a long time ago, China made, you know, garbage and we would break and you have to go buy new stuff. They actually make some pretty, you know, fairly high quality gear now. Yeah, I'm familiar from the EV space that that story is no longer, it's story no longer applies. Yeah, so by and large, a lot of the stuff we have out there, it's gonna work, it's gonna work as designed. It's got, you know, it's gonna fit within like the engineering specs that it says it'll do. So with that, is there a way to basically live on a diet of poison fruit? And that's really kind of the approach we're gonna have to take is what if we needed to, I don't know, repurpose this, you know, so we can take out some of the Chinese, you know, software components or firmware components and put our own on it. That's an option we've got where we could just basically just say thank you for the equipment. We're gonna repurpose it to our own needs and we're gonna put our own software in firmware on it and run it that way. Or we can force anything sold wherever it comes from to just have certain degrees of transparency, we can inspect it. And that way we can sample batches and say, okay, you know, X number of them to a high degree of certainty, we think are not tainted with some sort of bad code. Right. So there's different avenues that we're trying to push. None of them are, you know, like a switch. None of them are easy. None of them just, you know, roll out and make everything make the problem go away. Well, hasn't the Trump administration discussed to discuss the wholesale ban of these, which I assume is like, absolutely a bad idea. Not a good idea. Yeah, not a good idea. No, we, I mean, what we typically do now is when we buy this gear and we don't have these assurances, we just isolate the heck out of it so that it just can't talk to anything. Like it doesn't get to talk to anything except through this very slim channel and we inspect every single thing that it does says you over monitor it. So the just in case something did go wrong, you'd know. So you can't prevent it from happening, but you can certainly detect it. I mean, you can prevent a lot. You just can't prevent everything. But what you can limit it down to and then you just detect on that narrow band of what's actually allowed and it gets you a little bit more assurance that things are gonna be okay. I guess I just asked flat out, like, do we have reason to think that China wants to or is trying to do anything in the various with these inverters? Like just flat out. Do we have evidence, not just sort of vague suspicions, but do we have evidence that they are want to do are trying to get away with something? - Hey there, everybody. Don't worry, I'm not gonna tell you about a new mattress or push a credit card on you. This isn't an ad. There are no ads on volts. It is supported entirely by listeners like you. Feeling delusionary for a second, I'd like to ask for your support. I started volts because we're all surrounded by depressing news about climate change and misinformation about clean energy. And it's never been more important to share the stories of the real people on the ground doing the real work of transition and all the ingenuity, encourage, and public spirit they are bringing to it. People are hungry to hear these stories, to learn from and find inspiration in them. I've heard from people who change majors or careers after hearing episodes of volts. People using it in classrooms and community groups, even state legislators who have passed bills inspired by specific episodes. Sharing these stories matters. It makes a difference. If you have found value in it and want to help me continue doing it, I hope you will join the community of paid subscribers at volts.wTF. It's about the cost of a cup of coffee a month. If you don't like subscriptions, you can make a one time contribution. Leave a review on Apple or Spotify or just tell a friend about volts. I am grateful for any and all support. If you're already a paid subscriber, thank you. And now, back to the show. Yeah, short answer is yes. There's a campaign called Volt Typhoon. And the one I mentioned earlier, where they broke into the communications network is called Salt Typhoon. So this is the Typhoon series of actors. Volt Typhoon was, and it's still ongoing. And it's not like it was one group, and then we sent them home and they went away. It's an active campaign that is still ongoing. And we've actually found them. And there's documented evidence of this. It's public record where you found them embedded in utilities and various different systems. And they have, like I said, they haven't done anything bad. What they do is they don't break in and come in the front door with a marching band behind them. They don't announce their presence. They don't drop ransomware. They sneak, they come in very quietly, and they immediately hide. And they don't even drop malware. So they're not using, once they're inside, they're not using malware. They're using the tools that already exist. So we call it living off the lamp. They're very good at this. And it's extremely hard to detect. But we've found a few cases where we can see what they've done. It's shocked everybody because we were quite surprised at how good they were first, but also how deep they'd gotten embedded. But yeah, so there's documented evidence that they're actively trying to do this. They have done it. They're still trying. But do we know why? Like what are they up to? What do they envision doing with that access? They haven't openly communicated this to us. So all we can say is our best guess is, ostensibly, it is to exert influence over us, whether there is a particular trade situation, whether there is a particular Taiwan situation, those kinds of things. So they've got us in a very tight spot if we ever decide to take action against them in some way, where they need to leverage that. But it does. seem like if they came out and said, hey, guess what? We have access to a bunch of your energy resources and we're gonna shut them down if you don't do X, Y, Z. That would amount to something like a declaration of warfare, like that would cause the US to freak the hell out, I would imagine, and do all kinds of crazy things. It was not be a small step to take. It's not like a trade, you know, that would not, it would be more than a trade dispute. Yeah, one would think. So far, we've found them in our communications, like literally almost every single communications backbone, all the major telecom providers, and they admitted it. They didn't just shy away from a knack like they didn't do it. No, they openly admitted it, and we didn't do an act of war then. So, I mean, you would think the answer to that question is yes, but maybe is probably the response, I'm not really sure. Wild, wild. I guess it would look very different if they actually did something. Right, I mean, the fact that they haven't done anything sort of, I guess, let's all this kind of simmer in the background. Yeah. Let's talk about data centers. This is something I did an episode recently, you know, NERC, the Reliability Corporation, put out some alerts about data centers, just because the way data centers are built, they're very sensitive to perturbations, they're very sensitive to grid conditions, and they are prone to just switching off and dropping off the grid all of a sudden, which if you're a gigawatt data center, you know, gigawatt of demand, despairing all of a sudden can destabilize the grid quite badly. And so that was those worries were mainly about just the normal operation of these things, like the way they operate is somewhat of a threat, somewhat of a, you know, potential threat to the grid. But then it occurs to me as I'm researching this, like what if you caused that to happen on purpose? Like what if you got access to those systems and made that happen on purpose, you could destabilize grids via data centers. So is the profusion of large data centers is this yet another sort of attack surface and is anyone doing anything about it? Yes, that what's called a level three NERC alert for those that want to look it up, it was for what's called the computational load entity. And we've looked at these things, and this has been an issue in some areas, Texas is another example. Again, they have a lot of Bitcoin farms and they would basically automatically shut off so they would just absolutely draw enormous amounts of power until Bitcoin prices or whatever coin they were mining, whatever thing they're doing, whatever crypto situation they're in would hit a certain price threshold and they would just drop and it would just swing their grid hard. So they've put some rules in place there and then we saw some other situations that I mean recently was the one in PJM, a data center dropped three gigawatts. Good lord of power at once and swam. They can typically stabilize these things. I mean, most disturbance is we can stabilize them within seconds to, you know, few minutes at best. This took 10 minutes to stabilize, which is a very long time in power curves. Yeah, this is unprecedented. This is not something that has happened in history. This is a new thing in the world. Yeah, so when we look at these things, as I mentioned to IBRs, we look at it not just from a, we start with the grid physics components and we look at this from an all hazards perspective. So cyber threat, backhoe, whatever the threat is, if that system drops or comes back online and starts drawing and we didn't expect it, those things can cause grid problems. So we take a look at this from an all hazards perspective and cyber is one of those hazards and we're looking right now at how do these computational load entities affect the grid and what do we need to do to figure out how to balance this? 'Cause typically you have to have some sort of reserve capability or ride through capability, other like grid physics components, whether it's like static far compensators and their larger grid components that can absorb these swings and kind of help us ride it out without like you flopping off a big area just because you're having a disturbance. Right, didn't Texas just pass some ride through standards just recently? Yeah, they did. And we're looking at a bunch of ride through stuff for the NERC standards as well. So those would also guard against a deliberate malicious taking those things off the grid? Yeah, intentionally by design. Like I say, when we're, I've been on the standards writing process with NERC for 20 something years now. We look at it from, we call it all hazards, whether it's a cyber threat, whether it's a squirrel, whether it's a backhoe, you name it. We are earthquake flood. What happens to the grid physics and how do we actually stop that from being a problem? Interesting, interesting. Here's sort of like a broader question. One of the other things I was worried about as I sort of approached this whole subject and I contemplated the sort of, as I said, the sort of incredibly distributed attack surface we are developing on the grid. Yeah. Which is that there's got to be, just intuitively, there's got to be some trade off here between sort of security and speed. Like we, you know, we want to decarbonize the grid quickly. We want to electrify quickly. There's a great deal of urgency behind doing that as quickly as possible. And I just think intuitively, like the more standards you put in place, the more requirements you put in place, the more personnel are involved, the more audits are involved, et cetera, et cetera. That is inevitably going to slow things down somewhat. How do you think about that trade off? I also speaking as a recovering regulator. I'm a big fan of regulating only absolutely the bare minimum that you have to regulate and then let other forces operate above that threshold. So you will see things like business agreements. If you're an unreliable partner, you're probably not going to get a lot of business. If you're trying to grow a bunch of, you know, build a bunch of assets and grow a bunch of business, your insurance company is not going to want to do business with you and you're not going to get insurance for your plans. If you can't show that you've got some degree of responsibility in the mix. So there's other forces that push these components around a bit too. So just sunlight, just exposing who's doing what you think does some of the work. Well, it's going to be hard to hide. Because, you know, if a disturbance happens, we typically know where it came from. I like in the Spain blackout, for example, one of the bigger problems they had, they were able to come to a fairly rock solid conclusion. But there were still little gaps here and there because some entities just said, "No, we're not going to give you our data." And some entities just said, "We don't have the data. We don't log it." But in the US, you're kind of required to keep a lot of data. And, you know, we use a lot of data on our, just our operations and we even like resell a lot of our data to third parties and that kind of things for being a data broker, an operational data broker. So we have a lot of data. If something happens, we're going to be able to reconstruct that. And honestly, if a disturbance or a blackout happens, FERC is going to show up and it's not going to matter. You're going to have to prove it and show what you did. And you're going to get, you know, you're going to get that black eye or you're going to get the stigma for operating, you know, rogue or, you know, irresponsibly. So there are some things that will help. Are they perfect solutions? No, does it prevent it from happening? No. But every time we've tried to prevent a problem through regulation before it happens, there's a lot of rightful pushback that says, "Prove it. Show me that this is a real problem." So until we have, unfortunately, it's always a rear-view mirror approach. Yeah. And when you say you're going to find who's responsible, are we talking about the entity running the power plant, the manufacturer who sent the equipment, the software company that wrote the software or some of each, like who's typically, who are you going to find that was at fault if something goes wrong? The answer to that is yes. And for different reasons, right? Like if you caused a problem where my insurance company is trying to prove that it wasn't us, but it was somebody else, they're likely going to send in some forensic teams to go figure out what happened. And if there's software liability or hardware liability, they're going to chase that liability chain as far as they possibly can. Because hell have no theory like an insurance company that's been sworn. So that'll happen. Furkel come in, they'll want to do typically like a 1B investigation, especially if it causes a blackout. They send in a blackout investigation team. And those were some solid engineers that go to that work. They're good. That's like an NTSB investigation. So from all different angles, there will be sunlight on that event. Okay, well, this again, says to me from my next question really well, which is I think one of the sort of like disaster movie scenarios that people have in their head is that you get access in one little spot and then there's some sort of cascade. And you know, next thing you know, the whole East Coast is out or the whole country's out or whatever. And so I know a lot of your work and a lot of focus has gone into thinking about just how would you contain an event if it happened? Like how do you, and you call this cyber informed? engineering. So just that that's like what how do you build your devices in your systems so that things don't cascade basically I think is the simple way to put it. So talk a little bit about what that means. Who's responsible for that and what does it look like? Sure. This is putting in some we'll call them analog safeguards. So we put in some things that just aren't digital. They're just literally an electro mechanical thing and we used to have like a lecture mechanical relays and protection systems and we're looking to putting those things back in. Now they don't operate with the same level of like precision that we can do with digital stuff like a PNU or a phaser measurement unit, unbelievable precision in terms of what it can protect. An old lecture mechanical one, not nearly close to that, you know, it sure does come in handy if someone hacked the system, right? So if that's all you've got, it's better than nothing I would imagine. Right. So we're looking at those kind of things and a lot of other just kind of really physics-based behaviors. So like one of the best examples, I go back to like a water system because it's easy to visualize where electrons just aren't as easy. So like if the concern is too much chlorine getting dumped into the water because someone hacked the system and said, you know, put all the chlorine in right now. What we've done is let's make the pipe that delivers the chlorine super, super tiny. So even if you set the, you know, five billion parts per million, it just couldn't do it. Let's make the pump a little tiny pump that can only deliver so much at once. Let's make the battery supply for that pump only operate at a certain amount, right? So the pump just can't over, you know, can't over pump, right? And the reservoir that feeds it, let's make it really super small. So there's ways to do this, you know, now correlate that of the grid physics size side. There's ways to put in these analog gates and checkpoints where, you know, key areas are key, whether it's, you know, generation resources or large transmission through, throughput. We can do this on that scale and just, and it's relatively inexpensive to put these things in and it gives us a lot kind of a safety underpinning. So think of like a float valve that operates based on pressure versus a digital sensor. So we've got, we're looking at those approaches, but, you know, you have to remember, we don't have one grid. We've got four. Yes. Yes. And each distribution grid in some sense is its own little thing. Is its own, right? So I mean, we got the HVDC, sorry, ACDC ties that break up all the grids. So you couldn't just black out the whole country at once or all of North America at once. That's just not a reality despite movies. And then, I mean, when I get the conversation around, could they just hack all the things? And I'm like, if you really went out and did like an asset inventory of just all the things that are out there in the grid right now, even the really important ones and you just got those, the number of different devices would absolutely blow your mind. Like it's an incomprehensible number. Yes. Their ability to hack that diversity of systems is just, it's impossible. Well, it's, it's astronomically improbable. I won't call it impossible. Even with AI, it's just improbable. So, could they do key points and cause regional problems? Yeah, that's, that's a concern. But nationwide or continent scale blackouts are just not a realistic thing. And so when you talk about these physical, these analog sort of gate, gate limiters and type things, is that something that a manufacturer responsibility? Is that at the device level or is that at the system level? Is the utility the one doing that? Who's overseeing that? Yeah, this is typically the utility. Yeah, this is typically the utility that puts this in so that they can basically protect parts of their system that are very difficult to replace. Because reality, one of the bigger concerns we've got is, I'll keep it simple. The what are called protection systems relays. They're designed to keep the grid from burning down, basically like no one's certain terms, like literally burning down. So this is keep transformers from exploding all this kind of stuff. If we can, if we can put those for example, an electromagnetic or dumb relay on some of these really expensive transformers and other really expensive piece of equipment, even if they did hack it, this piece of equipment is going to keep it protected. So those are the kind of things that we're looking at. Really big generators, for example, obviously, you can, there's, there's ways to desync and re-sync a really big generator. You can actually torque the shaft and there's even some that have modeled like actually getting the generator to go boom. That's a problem. So you put these kinds of plates, these devices placed in the right spot there to keep the generator safe as well. So this is to keep our big components that take a long lead time to build that are really difficult to replace or fix. We're protecting those critical components with this kind of protection. Got it. So even if someone got in, cause some event, we have some reasonable confidence that it would remain reasonably localized. Yes. I mean, now, like I say, is there still a chance for problems? Sure. And is every big utility doing this for their critical stuff? Some are some aren't. So, you know, if there's no rule to do this, no requirement to do this yet, but it's those that I think most of them are doing it out of the financial incentive to replacing that transformer is going to cost us, you know, how many hundreds of millions of dollars and a minimum of a two to three year lead time. And now we're up against, you know, Google and Facebook and Amazon. Yeah. It's quite changing up in front of us. So we can't even get in the front of the line if we wanted to. So yeah, I think most of them are realizing this is just a good business decision too. Right. Well, once again, you've teed me up for my next question, which is about AI. And of course, when everybody, when anybody thinks about computing these days, that comes up, which is, which is that, you know, maybe the number and diversity of devices would be daunting for any sort of human hacker. But of course, now we've got these giant super intelligent robots who can just go at it without pause. So in your mind, does AI heighten the threat here? And then, conversely, on the other side, or is anyone putting AI to the task of building better defenses? What is the current role of AI in all this? Well, that's another podcast in and of itself. Really. I'll keep it to the two key things. AI is really good at finding vulnerabilities. So we are concerned about its ability to find vulnerabilities and key equipment that we're concerned about. So yes, it will definitely accelerate that arms race. At the same time, we're using AI to patch those vulnerabilities. So the manufacturers can, well, they'll see, there's a problem and they'll write a patch for it. And we can, the challenge is this is like grid equipment. So we may have to do things like, okay, general public, are you okay if we have like a blackout window and we schedule it because we got a patch stuff because most of the time you can't just like take these systems offline. Yeah, like an operating system update. Yeah, you know, yeah, it got a reboot. It's enough of a pain in the butt on your on your personal computer, but if the whole grid has to do it. Yeah, if you got a reboot section of the grid, that's a problem. So we're looking at ways to try to work this into operations. Some of it's going to be redundancy or moving the power over to a certain line while you restart those systems and try to get build this into our operations and our behavior practice because we haven't done this yet, right? We had to figure out how to make this work. So we're in the process of that as an industry and obviously the manufacturers have definitely the light bulb is gone off and they're reacting and they're using AI to write patches for the vulnerabilities that the bad guys are finding. Now the next piece is one of the bigger concerns that is not, we don't have an answer for yet. It's the ability to model the grid was pretty difficult in the past. It took a lot of computing horsepower. AI is actually really good at this and you can model large sections of grid. If not the whole grid, you can overlay other infrastructures like transportation and communications and gas and the ability to cause a multi infrastructure cascade by key component attacks in just the right number and frequency basically like the sequence of infrastructures to cause a larger problem. So we're worried about that. It's being discussed. It hasn't really like we're trying to model it ourselves and figure out oh, you know, if they're doing it, we should probably be looking at these same kind of key weaknesses and how do you get these cascading interdependencies down? This kind of stuff. So the thing that started is DHS Sissa CISA put out what's called CI fortify. This is the cybersecurity and infrastructure security agency. I wrote that down so that I wouldn't have to remember. I love how it's got security and it's named twice. That's really secure. But they put out one called CI fortify. The goal is at least if there's a problem to get these and it's not a requirement yet, the system is a suggestion. But to have these environments disconnect just what does it take for you to island to yourself off and we, you know, in the business we call it turtle mode. How do you like to plan down the shell until the danger passes and you peak open and you see the things are okay and then you can you know, pop your arms and legs out and keep going. So it's trying to get owners and operators of all the different infrastructures to go do an exercise on what's a day without connectivity look like? What's a week look like? Can you can you lock down in turtle mode and still operating keep essential services running even if not everything is working but enough is. So we're we're at least trying some approaches. So these These are like water systems. transportation systems, gas, electric, but are all the gas systems, man, that's a whole, that's a whole new Pandora's box right there. Trying to get multiple infrastructures to coordinate their behaviors, just seems like it's a whole new layer. - Well, if they can even do it on their own, great. Well, let's just start with them, figure this out what it looks like in your own little environment. And then we'll look at what it does when in terms of the interdependencies that cascade out. But yeah, I mean, think of like the dependence of electricity on the gas, right? We have a lot of gas generators, especially now for AI. There's almost all that we're gonna be gas generators. So it's amazing to try to think about this and wrap your head around it. But like I said, that's something we are at least discussing. We're starting with, you know, at the asset on our level, could you do this, could you operate? And then we'll figure out what is, what now, when you're doing it, what are the inter-connectivity, what's the interdependency issues, and then we'll take kind of take it out from there. But it's on our minds. It's interesting. It's hard not to, you know, it's hard to hear about this stuff and not get a little sci-fi dystopian about it, about a future where it's sort of like our massive super intelligent AI's are just battling their massive super intelligent AI's. And at some point, like it's gonna be hard for any human to even really know what's going on anymore to wrap their head around any of it, you know? Like eventually it's just gonna be the robots. Yeah, the robots battling over our infrastructure. That's why we're trying to put in a lot of that cyber-informed engineering underpinning so that, you know, even if the cyber causes problems, physics is still physics, you know? A couple of final questions. One, I'm just curious what one is, and maybe this is naive to even ask it, but like, are we trying to sneak, you know, are we trying to sneak stuff into China's systems too? I mean, I kind of assume we are. I mean, if we're not, then shame on us. (laughing) I mean, I really think those are kind of table stakes for nation states at this point, so. And we're remarkably capable at what we do. I mean, we're arguably the best in the world at cyber anything. So one would have to believe that that's a reality, yeah? It's just gonna seem like all these forces are pushing toward, you know, kind of reversing the trend of the last century, almost towards greater trade and greater independence, or interdependence, like it seems like this is gonna wanna cause countries to want to domestically manufacture everything so that they can keep an eye on everything, you know what I mean? Yeah, I think there's a push for that right now. There's a lot of bulkization of manufacturing, of, you know, sovereignty of various things. What we've seen in most cases that actually solves the problem, and I'm working on various different channels, I've got operations in Europe as well. It's about its transparency, so that there are open frameworks and everyone operates against the framework and everyone can see what's in there in transparency. Think about what happened with encryption. There used to be like, you know, export rules on certain types of encryption. Well, now we just make it so that it's, you know, it's open, everyone can see it. And it gets all the transparency that's needed, and it still works. So there are ways to do this where you have transparency into how it's done, what's being done, and everyone can agree upon kind of what the standards are. And then once, and there's, you know, IEEE, there's IEC, there are bodies that do this already and do it very well. So yeah, I think it's gonna, well, we'll have a push for everyone goes into their little hole for a while and realizes, okay, we can't do this. We just can't survive like this. So you depend on them, we'll swing, and then it will have to swing back. And all of this that are pushing depend on them back to the direction, we're all going, look, you know the solution to this is transparency. So let's just start there now. - Interesting. Another question is, when you read, or at least when I've read about attacks on the grid that actually do damage, its squirrels, or like red necks with rifles, shooting at transformers, like how do you rate the physical threats to the grid relative to the cyber security stuff? - Orders of magnitude worse than the cyber. - Interesting. - Orders of magnitude without question. - Interesting. - Yeah, absolutely. It's been, it's largely squirrels, raccoons, snakes, scorpions, I've seen a whole bunch of different things. Mylar balloons believe it or not, are actually a bigger threat than cyber. (laughing) So, you know. Has anybody weaponized mylar balloons yet? - No, no, but there was, there was like a Twitter account called mylar squirrel, and it would track all the attacks on just things that happened to the grid from like squirrels to balloons to raccoons to, yeah, I think it stopped, but it's far, far bigger issue. I mean, it causes a lot more damage than cyber. - And you think the vulnerability is greater? Like, I mean, that's what's-- - From the physical standpoint? Oh, absolutely. Yeah. I mean, if you can physically damp, well, that's what I would mention earlier, the physical protections for big transmission in the SIP standards, there's a specific standard for that. Like, you actually have to protect the transformers from gunshots. Like, you have to put up ballistic protection for the transformers. Yeah, because they're, I mean, that's a bigger threat. - Yeah, I wonder, that's another one that just seems like, as the grid distributes, as, you know, everything moves outward again, you're just like, you're not gonna get every warehouse to hire armed guards and, you know, put up ballistic fences like again, you're just a taxorface is so huge. - It is. One of the good things about that is if you distribute the attack service to a much, you know, more diffuse set of assets, you actually take the threat away in a lot of cases, 'cause they physically can't get everywhere. Any single attack will have smaller consequences, I guess. - Yep, yep. - Okay, so let's wrap up with this question. Right now, the aforementioned CISA, the cyber security and infrastructure security agency, is getting cut much like, much of the rest of the federal government is getting slashed, it's budget and staff. I'm just curious, you're sort of overall assessment. You've written an essay actually saying that critical infrastructure cyber regulation is headed in the wrong direction. I'm sort of curious, you're overall assessment. Like, are we doing this correctly and what is, if not sort of broadly speaking, what is the course correction you would like to see? - Ooh. - Again, a whole other pod if we-- - Yeah, and that is a whole other pod. I'll keep it short. I do lament the cuts to CISA. It is, American infrastructure needs this desperately. We've got a mix of things, the regulators are all different. We need something that is a centralized, an a harmonized effort that helps all of them. And most of what they've done so far has been fantastic outreach. They've done some really good information products that have helped a lot of people. They raised the floor for a lot of those asset owners that you mentioned that just don't, they can't hire a security team, but they can't take this template and apply it. Because it's free from CISA. And CISA did the work for you and you don't have to wonder, is this gonna work? You've got an authoritative source to get good stuff that helps you and raises the floor. So I would love to see more funding in CISA. I would love to see more activity from CISA. I don't want them to become a regulatory agency, but I love what they do with, there are no exploited vulnerabilities list, which this is a different thing. When you hear of a vulnerability that happens, Microsoft says, "Oh, we have a problem. "Go do something." What CISA does is if no one is exploiting this, okay, probably lower priority, but the minute someone's actually like attackers are actually using this, it goes on this special list that CISA maintains. That gives us the priority and because when you've got 100 vulnerabilities to deal with, which ones matter? This tells you. So that kind of work is absolutely useful and they need to be doing more of that. Now, the follow on of where should we go and if I had one rule that I could change and I get this question as a former regulator and someone who writes a lot of regulation all around the world at this point, I like the CIRCIA, it is CIRCIA, it's an incident response reporting act that was put out. It's gotten a lot of negative attention 'cause people don't wanna report their incidents and I get it, no one wants to air their dirty laundry. - Right, right, yeah, I guess it's embarrassing and. - It's embarrassing, right? And in some cases, you gotta put it on your 10K right now and you gotta report to the SEC, but people get around that and their lawyers step in and they water it down. My issue with this is we don't know what is working, we don't know what the attackers are doing, we don't have any actuarial data without this kind of reporting. We need something there. And I'm not saying, like, make them divulge everything, but we should have at least some data that we get that's useful, like a handful of things because the corollary is if, you know, we know that if I, you know, eat bacon cheeseburgers every day and I smoke and I never exercise and I never move, you know, Patrick's likely to die at age 45 of these conditions because we have like, you know, hundreds of years of actuarial health data. We don't have anything like, we're basically guessing. So without this, what is happening, what's working, we don't know where to put our money in our protections. I think this would actually really benefit us. If we could do one thing, we need to see what's actually happening. Because it helps us make regulation. It would be, it's a federal rule. - Is that a federal rule? - Yeah, it's a federal rule for critical infrastructures and there's, you know, a handful of them that get roped in first, that kind of thing. But we don't, we don't. So right now it could happen and if it wasn't a big enough issue, they could just hide it. And I get companies may want to do that, but the rest of the industry and the rest of the populace, we need this data to know where we're working and what's not. It helps regulators, it helps insurance, it just helps everybody all around. And when you say that cyber regulation is headed in the wrong direction, what do you think we're, what do you mean by that? What do you think we're doing wrong? I think we are, we need to harmonize a lot of our regulations like each individual infrastructure has their own, you know, authoritative body and they're all going in different directions. They've all got different penalty structures. They've all got, if you're a company that owns a gas infrastructure and an electric infrastructure or an electric and a water infrastructure, you got different and entirely different compliance organizations and different experts, the software companies have to write different versions of the software. I mean, we could standardize a lot of this stuff because realistically, I mean, it's this is all the same stuff. It's flow control, whether it's gas, electric, water, transportation, it's all flow control. So we're kind of doing the same things. The technologies are all roughly the same. We could make this a lot easier for a lot more organizations and then we could even introduce easier guidance. So we wouldn't have guidance for just like the gas operators or the electric operators. We'd have guidance. So we need to get to a place where we're not doing this in such weird silos and I get there are differences. Don't get me wrong. But there's a core component of this outside of those fairly small number of differences that could be done in a uniform way with a lot less wasted time and effort and money. Interesting. All right. Well, this has been so fascinating. Thank you so much. Thank you so much, Patrick. I feel like 100% better grounded at this point. Thanks for walking us through it. Sure. Anytime. Thanks, everybody. You've been listening to Voltz, founded and hosted by me, David Roberts, produced by Nate Peevy, and supported entirely through the generosity of listeners like you. If you enjoyed this conversation, please consider telling a friend about Voltz. That's how this show grows and survives through Word of mouth. See you next time.

Podcast Summary

Key Points:

  1. The US electricity grid faces growing cybersecurity threats as it becomes more digital and interconnected, with most power electronics sourced from China, though no direct US grid attacks have caused damage yet.
  2. Attacks on grids have occurred in Ukraine and Poland, attributed to Russian state actors, serving as proxy warfare and influence tools.
  3. A key distinction exists between IT (information technology) and OT (operational technology); OT devices, which control physical processes like breakers and voltage sensors, are harder to protect due to their design and need for constant uptime.
  4. NERC CIP standards apply only to the bulk power system (transmission and large generation), requiring measures like firewalls, physical security, access controls, incident response plans, and supply chain security, with enforcement by FERC-delegated agencies and penalties up to $1.8 million per day per violation.
  5. The growing distribution grid—including rooftop solar, batteries, EV chargers, and virtual power plants—falls outside these standards, creating a larger attack surface and regulatory challenges across 50 different states.
  6. Inverter-based resource disturbances in Texas and California, while accidental, have raised concerns about potential hacking, prompting NERC to study and address these risks.

Summary:

The conversation between host David Roberts and cybersecurity expert Patrick Miller explores the real and exaggerated threats to the US electricity grid. Miller clarifies that while there is documented evidence of hackers breaking into US grid systems, they have not yet caused direct damage like blackouts; such attacks have occurred in Ukraine and Poland, perpetrated by Russian state actors. A core theme is the distinction between IT and OT systems, where OT devices—the physical interface to grid operations—are uniquely vulnerable because they were not designed for connectivity and cannot easily be rebooted without risking outages.

The NERC CIP standards, which Miller helped create, mandate rigorous protections for the bulk power system, including firewalls, physical security, and supply chain checks, enforced with substantial fines. However, these standards do not cover the rapidly expanding distribution grid, which includes distributed energy resources like rooftop solar and virtual power plants. This creates a fragmented regulatory landscape across states, where the attack surface is exponentially larger.

Recent accidental disturbances from inverter-based resources in Texas and California have heightened concerns about potential malicious exploitation, leading NERC to study these risks. Miller emphasizes the need for unique isolation strategies and proactive management to address these emerging vulnerabilities, balancing between hype and complacency.

FAQs

IT (information technology) deals with corporate systems like email, data, and billing, while OT (operational technology) interfaces directly with physical grid equipment, such as sensors and breakers. OT devices are purpose-built and were not originally designed for connectivity, making them uniquely challenging to secure.

Yes, there have been attacks causing direct damage, such as blackouts, in Ukraine and Poland, likely by Russian state actors. In the US, attackers have broken into systems and prepositioned themselves, but no direct damage or blackouts have been publicly documented yet.

The NERC CIP (Critical Infrastructure Protection) standards are mandatory, enforceable rules for the bulk power grid in North America. They require utilities to identify critical systems, implement security policies, use firewalls, physically secure equipment, train staff, and have incident response plans, among other measures.

Enforcement authority comes from FERC (Federal Energy Regulatory Commission), which delegates to NERC (North American Electric Reliability Corporation). NERC's six regions audit utilities, and penalties can be up to $1.8 million per day per violation, though settlements often incentivize improvements instead.

Distributed assets, such as inverter-based resources in virtual power plants, are outside the NERC CIP perimeter and have a much larger attack surface. They are managed across 50 different state regulatory bodies, making standardized security approaches challenging, and grid physics issues have raised concerns about potential hacking.

No, but the threshold is being lowered. Currently, generators above 75 MVA must register with NERC, but this will drop to 20 MVA, bringing more smaller generators into scope because they have had impacts on the grid.

Chat with AI

Loading...

Pro features

Go deeper with this episode

Unlock creator-grade tools that turn any transcript into show notes and subtitle files.