Bug count sounds impressive... but means little

Hey everyone, I'm James Wilson and welcome to this solo episode of Risky Business Features. In this episode, I'm going to be talking about these patches and software updates that are increasingly coming out with massive bug fixes, massive change counts, massive release notes, it seems like the focus at the moment is how many things were fixed and any given software update. You know, just recently we've seen Microsoft hit, I think it was a thousand or more bugs addressed in a patch Tuesday. Oracle followed that up with, I think, 800 bugs fixed in a patch and they've doubled the frequency of their updates. Apple, the 26.6 release, there was a month or two ago now. That had 300 fixes in it, which was kind of an unprecedented thing for an Apple release, but the purpose of this episode is to call attention to the fact that counting the number of bugs and even worse becoming fixated on that and thinking of that as a measure of good of quality of progress is misleading. I wouldn't say that the bug count is useless, but I definitely think that first of all just looking at that opaque number of, or I guess I should say, not particularly informative number of how many bugs were addressed in a software update, that doesn't really give you what you need to make a judgment about whether that patch is suitable for you. How quickly should you be rolling out that patch? What are the potential risks? How much testing do you need to do? All of the calculus that would go on within an organization once that patch is available is very difficult to navigate when there is an overfixation of, or even the only thing available to you is the number of bugs fixed and perhaps release notes with, I don't know, if you ever looked at an Apple release notes, you can get five fifths of absolutely nothing useful out of them. Some vendors are better at it. But if that's all you've got, if all you've got is the release note in the bug count, that's not enough. So in this episode, I'm going to explore this from, I'd say four different angles. First of all, we're going to look at my association that not all bugs are equal. And in fact, it's about the platform and the runtime and some other factors is that determine the exploitability or the, I guess, inherent danger of any particular bug. The second point that I'm going to talk you through is I don't think, well, one of the other problems, I should say, with the notion of just focusing on a bug count is two things. First of all, there's often a huge number of bugs that are fixed and addressed that are then not part of that bug tally or not part of the security updates because they were found and fixed internally. But more to the point, the bug count is not going to be a place that necessarily reflects the oversized upside that you get from a single feature that might render an entire class or a large number of bugs inert, even if they're not fixed. And then thirdly, we're going to talk about how more fixes equals more change equals more entropy equals much more potential for regressions. And, you know, to me, even just the notion of a patch coming out that contains a thousand fixes, the other is really my, my initial sort of reaction to that is that's irresponsible. Unless the vast majority of those are exploitable or have the potential to be exploited or will be amazing quality of life or security improvements for the vast majority of users, there's just no defensible reason for shipping such a massive amount of change in a patch update. You know, it's one thing to have a major OS release that has a lot of change in it and people can, you know, make their own assessments about when do upgrade. But we're now jamming the same amount or if not even more change into regular patch updates. And I do not think that is a good thing. But at the same time, I'll admit that if I was asked, well, what's the alternative given that the, you know, the AI era puts so much power in the hands of the attackers, well, I struggled to come up with another answer, but that doesn't, it doesn't change the fact that I think that there is a major problem in the way that we are currently approaching software updates and these large amounts of bug fixes going in, thanks to the AI driven volume apocalypse and bug apocalypse. And then the fourth item that I'm going to talk about is the fact that as I said at the start of this, it's not that the bug counts on the number of changes that have gone into a patch is useless. It does have a use. It's actually quite useful for attackers or security researchers to map out where effort is being applied. Now, that's a signal that you could say you could always sort of determine from a patch and seeing where their changes are being applied. But it's almost like we've gone from a small sort of small radar beacon in a large forest to now a multi array incredible industrial strength radar setup that is just pinging signal everywhere all throughout an operating system or an application or a platform, which does allow you to sound out and understand where there might be over investment, under investment, where there's likely bugs to be remaining in an operating system or an application. And that, that, you know, is almost like a unintended second order benefit that these large bug count patches give to attackers. And then lastly, we're going to round out with, so what, what could we do here? I've got some ideas. They might be completely wrong, but suffice to say I do feel strongly that there needs to be a better way to be more transparent, report some more meaningful metrics and to equip the defenders, CSOs, CTOs, any sort of anyone in senior technology leadership with the real information that they need to determine how quickly and where they should be deploying patches versus just what we can do today with this so much noise about hundreds and thousands of bugs being fixed in any given patch. Okay, so let's start with the first reason why I think this fixation on bug counts is misleading. And that is that not all bugs are equal. In fact, it's the platform and the runtime and the environment that determines whether a bug is, okay, a logic issue, a potential buffer overflow or something that can be turned into a remote code execution to completely own a device. So recently I did some experimentation when Mac OS 26.6 came out with its 300 plus bug fixes, which again, phenomenal for an Apple point release. It got me wondering, what can I learn from this? What can I infer from this in terms of if there's 300 bugs that have been fixed in this release, where might there be more left over? And so using a combination of many different LLMs and my own custom harness, I essentially started off with the release notes for 26.6 and first said to the agent, I want you to go and catalog all of these bugs that have been fixed, then extrapolate from there on a couple of different dimensions including where a bug has been fixed, look for similar patterns or adjacent bugs that might still be there, test the completeness of the fix, was it closed off in one particular permutation? Could there be other ways to exercise the same vulnerability? But also take a look at parts of the operating system that have seen very little change and the life cycle of that part of the operating system, what code is it written in, what sort of privileges it has and build sort of a stacked ranked list of potential targets where perhaps there has been an underinvestment in security fixes. The results I got were really, really interesting. First of all, it was a thrill because wherever I looked, I could shake out more bugs across all of those dimensions. I found examples where the particular fix wasn't, like it might have fixed one particular entry point, but the underlying flaw was still present. There were examples of where there was literally something like an integer overflow in one particular function and then either upstream or downstream from that, a similar sort of thing that could be exploited. Well, I should say not exploited. We'll get to that in a second, but definitely a bug that could be demonstrated there. But some of the most interesting findings I got was from that examination of, let's look at parts of the operating system that are particularly old, that are written in less modern, less type-safe languages, with lots of, you know, mem-copy mem moves, all those sort of typical syncs. And combine that with looking for things that are quite privileged and have not seen a lot of fixes. Anywhere I looked, I could shake out bugs. And that is, you know, the wonder of working with these LLMs. And so admittedly, I got a little bit excited and then started to dream of how I was going to spend all of this amazing money that I was going to make through these bug bounty programs only to realize the actual reality of attempting to find portable exploits in a platform where the defense in depth has been the, I guess, the modus operandi for decades. It's really difficult. But it's not difficult because it's, well, let's be fair. I am at best a amateur security researcher, vulnerability researcher, but at least I can kind of hang my hat on being a pretty good software engineer. So this could be a skill issue, sure, but I'd like to think that my expertise, both having worked on a lot of different parts of macOS and iOS in the nearly 10 years that I was at Apple, combined with a long week experience in software engineering, gives me enough of an insider's to knowing kind of what I'm doing in the spirit of trying to find a bug that could result in something that is exploitable. And here's what I found, countless bugs and very, very few ways to turn those into usable exploits of any kind, right? Even just trying to get to the, you know, capture the flag system that Apple has built into the operating system for security reporting, you know, I could get quite a few to that point and have reported them, but turning things, turning these into and even chaining them together to create full end-to-end remote code execution or local privilege escalation or a sandbox escape, well, that has still evaded me weeks and months later, aside from one LP that did get reported, but that came from a different exercise. The reason for this is there's just so many different layers that you have to break through or overcome to turn any given primitive lack of, you know, a heap overflow, buffer overflow, a logic error into something that can actually get your arbitrary code running or escape a sandbox or achieve some sort of privilege escalation. But not all platforms have this. And this is why my assertion here is that if you just look at bug count or fix count in a software patch as an overall signal of, well, that's good, then you're really, really missing the point. Because I would wager that there is a massive difference in terms of what you can infer about the state of the platform, the potential for additional bugs to be there and to be exploitable as in, you know, real latent risk in a operating system that has many layers of defenses like macOS and iOS with things like memory integrity enforcement and, you know, read, in particular read only regions of memory in the kernel, you know, the page protection layers, modern run times like Swift, for example, the build in a lot of memory safety or all of these things that are present in a platform like macOS and iOS or what makes 300 bugs plus the ability to maybe shake out another 500, not really result in a huge amount of actual exploitable vulnerabilities that are still there. But conversely, if we're talking about something like a security appliance or a application that runs on top of another operating system where there might not be the ability to create those, you know, full vertical integrated security mechanisms, well, that's where a couple of hundred bugs in any given patch could actually be a real concern about the potential for other latent bugs to be exploitable and rapidly exploitable. You know, take, for example, and we've talked about this on the main show, the difference between something like the corona exploit kit or the dark sort of exploit kit where you had, you know, a primitive, a read, right primitive two separate primitives actually and webkit one for reading one for writing arbitrary memory that were then chained together to do sandbox escapes that were chained together to get into the kernel that relied on undocumented registers to be able to write to memory that the kernel would lay to write to all of these amazing steps that you had to go through to circumvent these layers of security versus something like a security, edge security device, let's call it, where a single buffer overflow does actually lead to remote code execution, right? That's the, I guess, the thing I'm trying to highlight here is that we shouldn't just look at 300 bugs fixed great 800 bug fixed even better 1000 bugs fixed even even better. That's just the wrong correlation. It's the wrong inference as to the bug count. What you need to be thinking about is how many bugs were fixed? Where were they fixed? What does that tell us about the potential for other bugs to be present? And given the combination of either the hardware or software or the runtime that is this operating system or is this application or where this application or appliance runs, what do we know about the potential for any given bug that is either not been fixed properly or still exist in the system to be rapidly turned into a remote code execution privilege escalation or something that is otherwise going to get that box owned and that just does not come through from the bug count alone. The second thing that in my mind makes this fixation on bug counts misleading is the release notes for any given software update or the fixed count that disproportionately focuses on small high volume of fixes versus a small number of very, very impactful changes whether it be features or mitigations that could be implemented that could render an entire class of bugs inert. And again, I'm going to come back to the example of Apple here in contrast to Microsoft where Apple has been incrementally rolling out over time more and more defense in depth. I think the memory integrity enforcement was one of the most recent ones where essentially areas of memory attacked in a certain way such as if a corruption does happen or an attempt to corrupt happens that that metadata helps to defend against that a mitigate against that. And so then again, you could have a bucket load of buffer overflows and heap writes and reads, but they're not going to turn into something that is immediately usable for you because there are these mitigations. You know, pack the point of authentication is another example just because you can corrupt memory to the point where you can control the the program control register and say, Hey, I now want you to jump to this instruction. What it doesn't matter on it doesn't matter on a platform that has packed because the, you know, that the kernel just crashes immediately because that signature is not in place. And therefore, again, what we're doing here is we're not removing all the cases where a particular latent bug could give an attack of the ability to change the control flow, but rather we're saying control flow is just simply not something that an attacker can manipulate because those control flows have to be signed. Now, of course, as with any mitigation, there is very creative ways around this and we even did see in Karuna a very, I mean, just deliciously good bypass for pack where essentially the just-in-time compiler within a webkit within JavaScript core that is responsible for signing that code that's being compiled within the JavaScript interpreter, those widgets were used essentially against pack to generate real signed valid packages. But like, you know, that that's chef kiss level of of working around something like that. My point here is just that if we get overly fixated on more bug fixes equals better, then we're going to, then essentially enterprises become overly incentivized to ramp up those bug fixes to take more bug submissions, to take more fixes. And, you know, that's not cheap for them. That is a huge drain on resources. Engineers, whether they are fixing bugs, reviewing the fixes to bugs, testing things, they're the same people that are going to be having to expand effort, whether they're working on features that could potentially mitigate an entire class bugs and render those inert even if they're not fixed versus having to go through and apply all these fixes, test all these fixes, reason with the bug reports, et cetera. And I get worried that we're heading into a situation where because just the the deluge of bug reports and the focus on them and the fear and panic around the bug apocalypse and vulnerabilities, distracts us from the fact that actually it would probably be much better to see a release come out on any given application platform that said good news. There's only one change in this release and there's one change renders this entire class of bugs inert and he is the mitigation strategy versus an update that says, look, he is 500 fixers and they all center around many different types of bugs. Are these going to be the only bugs? No, not a chance. There's still going to be heaps more. Grugs are the best. Infinity minus 270 is still infinity. It's that sort of contrast, I think, we need to keep in mind. It would be just so much better to see a small number of changes that result in broad spectrum mitigations as opposed to an unhealthy fixation on just take more fixes, more fixes equals better because surely less bugs equal safe. It just doesn't work that way. Of course, it's important to fix bugs when they are identified as being known as exploitable or have real tractable, provable paths to be exploited. Sure, I get it. You got to do that. But I guess what I'm arguing for here is a level of discipline being instilled that ensures that there is actually a stronger waiting placed on implementing, let's say, security strategies and features that mean that that while AI will continue to report hundreds or thousands of bugs in any given app or platform, the impact of those and the need to rapidly fix those and to introduce the entropy that comes with those fixes is actually less important because we've invested in the mitigations up front. Now, the other thing to say here is yeah, it's almost like no time like the present, but actually platforms that don't already have these mitigations in place are out of serious disadvantage here because they are the ones that are now struggling with the notion of perhaps we simply can't take the time now to sit down and work out how to come up with a great strategy for rendering entire swaths of bugs inert because we have the wolf at the door, we have the thousand bugs in the queue, all of which are being exploited in the wild and we're at some sort of existential crisis. Even Microsoft, I think, is struggling with this or I remember the story recently how they wanted to start turning on their version of something that's like memory integrity enforcement but because this is, I guess, an optional quality of the platform as opposed to being something that is just inherent to the platform on by default, then there's a difficulty there because of course not everyone's gonna turn it on, not everyone can turn it on because perhaps they can't deal with the performance implications of that. And so, I think it's an interesting point in time when it is worth taking, I guess, a look across the various platforms that are in use in your technology landscape and say, whether we like it or not, some platforms just have an inherent advantage at the moment because of the particular focus on security that they've had for a long time and/or the fact that they're not now scrambling to try to manage the trade-off of, do we fix the deluge of bugs coming in to just avoid the exploits that are coming at us versus can we still spend the time on these more proactive, longer-term security strategies that will over time have a much better, a provably better impact on creating a safer platform in their medium to long-term. Now, that trade-off of, do we keep taking the fixes or do we take a pause to work out what we've got to fix or that we don't need all these fixes, leads me to my third point around the misleading nature of bug counts. And that is, put simply, more fixes equals more change, equals more entropy, equals more regressions and regressions are not good. Let me take you inside the world of software engineering at Apple throughout the 2010s. Back then, we'd be working on usually, let's say two in-flight releases at any given point in time. There was the next major operating system and then there was the mid-year operating system updates. And you know, there's these ebden float a little bit over time, but typically we were working on one very large set of changes that was the next major operating system or NMOS. And then we were probably also doing a little bit of work to either backport some of those fixes into the mid-year release or to work on something specific that was a particularly meaty feature that was being released in that mid-year release. Either way though, those releases were structured into milestones. And typically, you know, let's say four milestones, the first milestone might be a month of work and then a bit of cleanup. And that's when there'd be a huge amount of change and the bills would be unstable. They wouldn't be all that usable from day to day. But then by the time you started working on the second milestone, the second month of development, the build had become more stable. You could often install, let's say, one build a week and just deal with the bugs. But the focus we're starting to shift towards, I guess, the quality of the platform, bug fixes and polish, et cetera. By the time we got to the third milestone, the third month of development work, the focus really was not on feature work anymore. And in fact, if you were still working on features, you often had to punt them at this stage so that you could just focus on bug fixes and polish. But at this stage, the builds day to day would be livable. They'd be usable. Module over a few regressions here and there, day to day, you'd come into work, install the new build, test things out, report the bugs. And we were all just focused on fixing bugs, improving the quality, getting ready to ship. Then the fourth milestone, again, another month of development work, that would be just entirely focused on performance work. And at that stage, you absolutely were forbidden from working on feature work. There was only, you were only taking bugs that had a very low risk profile and a good benefit. But overall, the focus was fixing performance or regressions and especially on older generations of hardware, eking out every bit of performance so that we didn't just ship something that worked great on the latest generation of hardware, but also worked on the previous years. When we got down to the tail end of the release cycle, those last couple of months, we would begin to enact a thing called the bug review board, a BRB. And this was quite an experience. The BRB would meet every day towards the end of the release schedule. And the builds would increasingly be locked down. There'd be a period of time where we would say, look, we're only taking priority two and worse fixes. And then after that point in time, we would declare, OK, there's no more P2s to be fixed. We're in zero P2s. We're now only fixing P1, priority one bugs. And then you'd get to a point where it's OK, there is no known priority one bugs, right? So that would be turned zero bugs. And at that point on, you'd only be fixing new things that would come to light. And they had to be incredibly bad bugs with an incredibly rock solid story around how you were going to ensure no regressions and test for regressions and ensure a really high quality of release. And anyone that actually wanted to check a change into the build had to front this bug review board, which was a big boardroom with all the senior leadership. And it was not a pleasant experience, folks. When you showed up, you would get grilled. There would be a huge amount of skepticism and a huge amount of almost like a proof to us why we should take this change. And as unpleasant as an experience, it was, it's the reason why Apple achieves a quality of its iOS releases. And say what you will about the quality these days. But I like to think that there was a time when we knew how to do this really, really well. And it came down to this discipline and this almost like categorical rejection of change that increasingly ramped up towards the end of a release cycle. Now, maybe it's just the underlying pathology of having lived through that year after year after year, a bit of PTSD, I don't know. But when I think of that experience and how difficult it was to get a last minute change put in the build and how galling it was sometimes to know that you were just going to have to ship something with a bad bug. And yes, the fix would come out in a month's time and a point release. But no one felt good about the fact that we shipped this thing with a bug at a given point in time. But that was sometimes the trade-off we had to make because the risk of taking a bug, taking a change, introducing a regression and not catching it before release, was worse than shipping something that we had a known problem that we could either document, as much as Apple documents these things, or at least be sure that we could arm Apple Care and whoever else needed to know about us to how to work around this in the worst case that it did actually impact customers. So when I think about that experience and I wash that against a patch Tuesday that has 1,000 fixes in it from Microsoft, that just doesn't sit right with me. And it doesn't sit right because I know that that is going to result in regressions. It's going to result in things either not being fixed properly or the fix itself introducing a bug or the fix introducing some timing issue or something else that exacerbates a race condition that's always been there. But now that crash that happened maybe once a month now happens every day. And that's already happening, right? We have seen already a couple of recent patches from Microsoft introduced some regressions in I think there was some bugs in audio, et cetera, right? Not necessarily the internet melting kind of things that we could potentially see. But one of the other things that I worked on at Apple was the software update infrastructure. And I'll tell you, it's things like taking 1,000 changes in a patch and the potential for there being a regression in something like the software update mechanism and that leaving people stranded on a build or there being extra steps required to install an update because the software update or auto update mechanism regressed and then a really serious, exploitable RCE bug being present in the release that they then can't ship an update for or can't get people to rapidly update. That's the nightmare scenario that I see as honestly inevitable when we're taking so much change so rapidly in these patch releases. It's kind of funny, we expect modern software to be stable, to be predictable. And honestly, that is an utterly unreasonable expectation given how complex the software and hardware stack is now. You know, you think about how many moving parts there are in any given single computer, single node where you've got. multiple cores. You've got parallel threads. You've got all of this complex scheduling and asynchronous behaviors happening. That's only at the software level. Think about what's happening in the kernel. Think about what's happening at the hardware level. Think about the frequencies that happen in the gigahertz and just so much happening in almost like this sheer cloud and noise of chaos. And yet somehow we hold this to a bar of, no, it must be absolutely stable, absolutely rock solid, absolutely predictable. It's unreasonable. It was unreasonable then. It's unreasonable now, and especially so given that we are taking so much change so rapidly in these software updates. The analogy I give you here folks is we expect software to be like the smooth surface of an absolutely still pond or lake. Think of that beautiful picturesque lake and the water surface is so, so smooth, so uninterrupted that you can see a perfect reflection of the mountains and the scenery, that's what we expect out of software today. Despite the fact that the reality is that what's under that surface of the water is this massive churning asynchronous complex machine. And yet now we are holding that expectation up a whole new level and expecting that same absolutely placid, smooth, uninterrupted, mirror-like reflection surface of the lake while simultaneously throwing handfuls, bagfuls of pebbles onto that surface of the water from seemingly every direction at once. And the reason I like this analogy not just because it creates a nice view in the mind, but any one of these changes that you take, the disrupts that surface of the lake, it creates those same concentric circles, those ripples along the water in the lake just as it does in the software stack. Like the number of times someone would show up to that bug review board and say, "Hey, I've got this really bad bug. It's okay, we've got a fix for it. It's just a one-line fix and you could hear the whole room at that point stop and collectively cringe for the barrage of abuse that this person was about to cop because they were about to get a very serious lecture about the fact that it doesn't matter if it's a one-line fix. That one-line fix could have untold ramifications throughout the rest of the software stack in ways that you can't possibly reason with." And I would say not even the most advanced LLMs today can reason with because that's just the nature of software. So again, this fixation on more bugs fixed, you know, more bug fixes in the patch equals good just doesn't hold up. What's that he's going to result in is more instability, more regressions. And the second order effect of this is that that puts downward pressure on CSOs and technology executives to delay patches. The moment you patch something, a critical system that has some massive impact on the organization and the post-incident investigation for it comes out with the root cause of, well, this outage with its, you know, millions of dollars of impact and hours of customer impact was all caused because of this software update that was deployed. The net effect of that is inescapable. There is a pressure on leadership that says, well, don't patch. Don't do that so often. Test it more. Delay the patch. And, you know, that's simply incompatible with the guidance that we received today, which is you need to simultaneously be patching more often as well as being careful of software supply chain risks, which requires you to patch less often. But my point here is we just can't think of more patches equals an overall net good. Not until we've solved the overall fundamental challenge of software engineering, which is regressions and the fact that our software stacks are so complex that any given change, anywhere in the system has a ripple effect and creates a perturbation that will cause other problems to come to life. And that, I don't see that being solved in any way short of AI and large language models coming up with an entirely new stack of something that sits on top of hardware that completely replaces the layers and layers and layers of abstraction that the software ecosystem and stack has become as we know it today. So, so far, I've talked about the shortcomings of bug counts and release notes as we know it, right? So, first of all, it said not all bugs are equal. Therefore, we shouldn't be just comparing the number of them. The count of bugs doesn't, you know, give the same sort of celebration to that one feature implementation that might render an entire class of bugs inert, even if they're not fixed. And thirdly, more fixes equals more change, more entropy, more regressions. Okay, so there's the shortcoming of the bug counts. But I said they're misleading and they're not useless. So, let's look at a way that they are useful. You know, at the end of the day, finding bugs, turning them into reportable findings or even exploits is still, I guess, a volume game, right? There's only so many parts of the operating system or a codebase that you can throw out a human researcher or even a large language model or a combination of the two before they become overwhelmed. And so, there's a triage needed to determine where should we go looking? Why would we look there? Why would we prioritize this over that? And my point here is that these large patches that are coming out helps an attacker navigate that because take, for example, what I did with the 26.6 release, I said to a model, go and look for parts of the operating system that are old, established, use non-memory safe implementations. So, older generation of components written in Objective C or better yet in straight up C. Look for parts of the operating system that have got a lot of memory calls in them or syscalls or are doing the kind of things that if we could corrupt or change the control flow would be particularly useful to us, then run that up against where there is a part of the operating system that is particularly privileged and then create me a prioritized list of things we could go after. Now, that's one thing, but then when I can add in the signal of N-prioritize, those parts of the system that did not receive a lot of patches in this recent update, well, that helps me in two ways. Not only does it give me the signal that this is a part of the operating system that potentially has those sorts of bugs and issues in it because no one has applied those fixes yet, but it also tells me that this might be an area where I'm the one that'll find the novelty because for whatever reason, it is not or has not yet been the focus of other researchers by way of the fact that we haven't seen a whole bunch of third-party reports resulting in fixes in that area. Okay, but so what? What can we actually do here? Rather than just tell you all the reasons where I think fixating on the bug count is the wrong thing to do, what's the so what here in terms of actual change that I want to see? Well, I've got a couple ideas. These might be naive, these might be completely wrong-headed, but I just thought I'd share them for, you know, start a conversation, put an idea out there, see what happens. The first thing I think we should look at is other metrics that might be interesting here. We are starting to see releases come out where there's a commentary of X number of bug fixes and how many of those are actually known to be exploitable or are known to be currently under active exploitation. That I think is a good signal. We need to see more of that. I would love to see that become just the norm that there is X number of bugs fixed or changes taken, and here's the ones that we know are resulting in actual exploitation in the wild now. That that seems par for the course everyone should be doing that. But I think just as larger language models are empowering everyone everywhere to find these bugs, we need the same sort of unlock in terms of using large language models to triage those bug reports, those vulnerabilities into is this likely to be exploitable or not? And how likely? Because ultimately what I really want to be able to do is to look at a patch and determine, okay, what's the potential risk here of not deploying this, not deploying it on an edge device, not deploying it on a workstation, not deploying it on an internal device. And in order to do that, I need to understand the potential for exploit. But I don't think that's coming through clearly enough when there's just a commentary of, here's all the bugs that we fixed, you work out if it's important or not. And here's a couple that are known to be exploited right now. We sort of need that probability of exploit coming through to be able to judge the risk here, especially as I said before, if we're facing that downward pressure that will come from a software update that has in the past caused a regression or a problem in a particular part of a system or an enterprise. Maybe another thing that would be interested is bugs that have been reported but not fixed and why they're not being fixed. You know, again, comes back to the fact that not all bugs are equal. I actually think it'd be quite interesting to have a commentary of, look, we received all of these bug reports and we've deliberately parked them in a not to be fixed queue because all of these are, you know, yes, they're bugs. Yes, if you've got a debugger attached, you can demonstrate that you've been up to corrupt the memory bit. All it does is result in a crash. And here's how we can prove that, you know, There is a high degree of confidence around the mitigation being in place for these things. Because again, you know, there is a different mindset for me anyway in terms of determining the urgency with which I need to deploy a patch. If I knew how those mitigations were functioning and the kind of bugs that they were being identified, but already provably rendered inert by certain elements of the platform. There's inherent qualities within the platform, so maybe that would be useful as well. All of this comes down to a call for more transparency, almost like Dera said, a radical transparency where we should not only be talking about the bugs that have been fixed, but we should be talking about what's been fixed and the potential for exploitation of it, as well as what we know about but are not fixing either because we haven't got to it yet. Or we just don't intend to get to it, and here's why we're comfortable with that because we believe that these mitigations are in place and here's how it works. That I think would be quite useful and get us a bit further ahead than just a fixation on bug count gets us at the moment. But the other thing I've been thinking of is, you know, what if there were different types of patches available? Right. We've gotten used to the fact that we can select the time horizon for patching, you know, in my Mac or my iOS devices, I can go into settings and say, "Hey, I want the developer betas, those are going to get me the most recent changes sooner." I can say I want the public betas, which will get me sort of the middle level, right? It's not just going straight to the developers, there's going to be some bug fixes applied, some assurance that this beta release is ready for the public. So, you know, it's not going to be as stable, but it's also going to get me access to the newer features, but with less risk. Or I can say, "Nope, I just want the release versions." And, you know, I tend to believe this is the norm on most platforms. There's some sort of concept of beta releases or advanced updates. So we've got comfortable with, I guess, giving the user a level of discretion of updates in terms of the time horizon, the particular, and how that might impact the quality of them and the risk of bugs, et cetera. So why can't we do the same, but on the dimension of the number of fixes that were actually taken in an update? Why can't we say, here is, you know, this security update, and there are three permutations of it. There's the patch that has all the fixes. There's the patch that has just the fixes that we believe are either being currently exploited and/or have a medium to high probability that they could be turned into exploits. And here's the patch that only contains the fixes for things that we know are being exploited right now. Now, you know, I'm not as naive as that might sound. I get that there is going to be complications in terms of sometimes you've got to take a set of fixes. Sometimes they can't be just separated out like that. I'm also mindful of the fact that the more, you know, different permutations of fixes, you have the more variety of patches, the harder and bigger the testing matrix becomes and it becomes a support nightmare when you've got potentially different configurations. Like I'm not advocating for something like the Linux packaging system here. Where, you know, God knows how many permutations of any given Linux system there are. I dare say it's larger than the grains of sand on the planet Earth. But I'm advocating for some degree of difference in any given release where a defender or someone in an enterprise can say, look, we do need to patch this system and it is a very, you know, high risk system and it is also something that has a potentially very high blast radius or impact for the organization if something goes wrong. And, you know, it's not on the edge of the network or it has, you know, certain other protections around it. So maybe for this one, we just take that patch that has the known exploited vulnerability fixes or maybe I can make a different decision about something that's on the edge of the network where I say, you know what? This might result in regressions, but I've just got to take the patch that has absolutely everything because I can't risk the chance that an exploit will be found even if it is in the category of what was thought to be a low probability of exploit. That's the level of control that we're not given when the only lever that we've got is, you know, patched with the beta patch with the public release beta patch with the release version or patch with the advanced fix, et cetera. The time horizon has now largely collapsed. We are moving to a point where we do have to patch now, as soon as the patch is available. So my question here is, should we change the fact that it's not whether the patch is available and at what point in time and with what quality of fixes, but rather say that look, here is the patch and here is the different permutations of fixes that are in each ranked by the known exploitability, probability for exploit or just general, you know, good hygiene fix to be taken and let the defender determine how to manage their own risk accordingly. Maybe that worked. I don't know. Suffice to say, though, that the entire way that we do software updates, I think, needs a thorough re-examination, especially in this AI era, it's already so thought with problems of different update mechanisms, those update mechanisms, mechanisms being hijacked for supply chain attacks, different signing capabilities, different discipline in that, and now we've got the added problem of, this deluge of bug fixes and trying to jam them all in and time before there's an exploit, something needs to change here. Thanks for listening folks. I've been James Wilson for Risky Business Features.