Demystifying the Top 5 Questions of AI Governance

You're listening to Data Science Leaders, the premier podcast for executives tackling the world's most important challenges using the power of machine learning. Let's dive in. In the AI world, we are talking a lot these days about AI governance and responsible AI, and it's more important than ever that you, as an AI leader, business leader, or data science leader, are equipped to participate in this discussion as it heavily impacts. Your ability to drive impact and adoption of AI machine learning and not to mention your career. So to help you in this episode, I as a professional AI mansplanner will equip you with opinionated answers to five of the most foundational questions about AI governance, really questions that all of us should be asking and be able to answer. So, diving in, question one, why should I care about AI governance? Oh, I mean, this might seem obvious, right? Well, we care about it for altruistic reasons because it's important for doing the right thing for ourselves, organizations, and society. And that is absolutely true, and good on you if that's the reason why you're doing it, but it really isn't enough. It's not enough to make this self-sustaining with an organization and to funnel the kinds of investments that we need. The regulation, a very key one, I believe, 45 states have introduced some kind of regulation, proposed it this year in the 2024 legal legislative session alone. And I think so far about 31 states have enacted some kind of regulation or resolution. In itself is also, it's important, especially if you're in the EU, the fines that could be accompanied are potentially astronomical, and in the US and in other countries, it's becoming more and more likely that you will be facing some regulation of some kind. So it is definitely important and becoming more important, but it's also just fundamentally important for performance of our AI and machine learning projects. The only way to ensure that your models are working as intended, are working reliably and robustly, is to ensure that they are governed appropriately during the development and deployment process. And that's the way that you yourself can trust that these models are performing correctly in this, AI governance, but it's also the way in which you drive that trust with the rest of the organization. AI governance is critically important to be driving trust and biotrust, adoption, getting things from POCs into production, and getting folks to use them in production. So really, AI governance is particularly important because, well, regulation, because it's the right thing to do, but also because it is in our very direct self interest. And to be able to drive impact and transformation with AI, we absolutely need AI governance. There's yet another reason, which is that there currently is a vacuum in most organizations around AI governance. There are a few leaders who have the background and experience and often the willingness to go in and take the lead on driving these. And we absolutely need leaders to be doing this. It is critical for the organization, but it is a great career opportunity for you yourself to be able to drive this, because it is a vacuum that needs to be filled. So either take advantage of it and drive this yourself, or go in and support other leaders who are willing and able and have the knowledge and background to do it. Question two, maybe this should have been a question one, what is AI governance and how does it relate to ethical AI, responsibly AI, trustworthy AI, explainability, interpretability, and so on. And the reality is, well, it isn't very clear. These are very overlapping terms. They haven't been defined very well by any central authority that we can all agree on. We often use interchangeably by folks that are out there today. And this is a shame. We have a huge problem with this in AI generally, and we need to get more specific in order to be able to drive progress here. We would recommend that you approach things like responsibly AI and trustworthy AI as the entire realm. This really covers anything and everything that you would be doing when it comes to either ethical AI, governance, explainability, et cetera. The people aspects of ensuring that models are behaving as we want them to it covers the process and technology elements of it and so on. It goes from both expands from principles to practices. But it is also too large when we use that term we are referring to it too much. So we can get more specific on the areas that we need to drive the most improvement on things like principles and AI oversight committees. This is what we're actually pretty good at. It's really getting to the brass tax of how we put this into production. What are the systematic activities that we should be doing to manage risk with our AI machine learning models. And that is how I recommend you to find AI governance. AI governance is where the rubber meets the road. It's the road. It is about actually translating principles, ideals into the practices of the organization. So turning those frameworks into specific activities that individuals and teams are taking on a daily basis in order to manage risk. Along the way there are other terms like ethical AI, ethical AI obviously still very important, but arguably only applies to narrower set of AI machine learning use cases that actually have ethical AI consequences. So while I do think it's important that we talk about ethical AI, when you're going in and implementing these in your organizations, I would recommend using a broader term because the same kind of considerations that apply to ethical use cases with ethical considerations also apply to use cases which have risks that are not of an ethics nature that aren't related to fairness and unfair discrimination. All of the things that we do in order to ensure fairness are things that also help us when we are looking at use cases that don't have that ethical component. So better to go in and focus your efforts either at the responsibility level, which is really, really large, but even better to focus it in on AI governance more particularly. That does also bring up the question of, well, OK, AI governance, if that's about putting into practice, if you're in finance, you might think, well, it doesn't this sound like model risk management that we've been doing for years. And I would argue, yes, there are lots and lots of similarities. We should be building on those hard one lessons that we have acquired through governing use cases like credit risk scoring use cases that have been heavily regulated that have very important ethical and fairness considerations. We should be building on that when we are developing our AI governance practices. But model risk management needs to evolve. It is definitely limited to traditional predictive models. And most financial services organizations that I know struggle with our MRM practices because they are so heavy, so manual, complex, and they'll be very difficult to go into apply to other use cases in the organization, which don't require as much oversight, which you don't have as much risk. So building from model risk management, but making it more flexible, making it more automated, which sort of brings us to question number three. Well, so why is this so difficult? Why don't we see more in the way of governance? Oh, it's difficult because so frequently governance is an afterthought. It's an add-on. You do all of your work. You go through this discovery process. You're not collecting evidence along the way. You get to the end point of it. And then all of a sudden you need to go back and collect that evidence. Go in and make the case, go and explain to others that you're doing. So many of their existing approaches to AI governance are tying a bow at the end of a project, very expensive, a manual, and a difficult bow, versus embedding governance by design into our processes so that this is something that we're doing on an ongoing basis. And the result of the fact that governance is often an afterthought is that a lot of the results that folks get, a lot of the models, are just not reproducible. So you find people going in and having to recreate the wheel, going back and trying to identify, okay, well, what were the data sets that were used? What was the code? Why doesn't the code work? What were the environments? What were the versions of the libraries that were used? All of that we need to make reproducible and auditable in a way that for the most part isn't today. Another challenge is that, well, it's often the policies are often either not clear. They haven't been formalized or that there are multiple policies that are conflicting and that aren't necessarily perfectly applicable. So we need to get a lot better about defining those policies, managing those policies, and adapting them where necessary. And there's also the issue of that often these policies and the processes are not really enforceable. We don't have the control mechanisms in a lot of organizations in order to go in and prevent risky activity from to prevent data that shouldn't be used getting used to go in and prevent models that haven't been tested going into into production. And we don't often usually have the visibility around these. There are, it's wonderful. There's a Cambrian explosion of projects around the organization. But who is it who has visibility into all of that? A lot of that, that visibility is a lot of the this goes without any oversight. And on top of all of this, there's the fact that everything is done manually. There's very little in the way of automation in most organizations right now. And that has barely worked so far when we were just talking about a handful of use cases and models that aren't being developed and maintained very frequently. It absolutely doesn't work when we're looking to drive organization-wide transformation with these technologies when we're going after a much, much broader set of use cases, and also a much broader, wider set of projects and technologies involved. So in order to be able to manage this, we as human beings just aren't equipped to do this manually. We need the assistance of as much in the way of technological automation as we can get. And as you can guess, the result of this challenge is that a lot of organizations are sitting on ticking time bombs in terms of that the models that they have deployed are-- if they fail, folks aren't going to be able to go back and figure out what was done. They're not going to be-- it's not going to be possible to be diagnosed. There are going to be fines. There are going to be business impact consequences of the models not performing that the way they should. But above all, there's also going to be a lot less adoption. You're not going to be able to take advantage of that Cambrian explosion of innovation in AI, because at the end of the day, people aren't going to trust what was built and developed. And I'm going to trust that it was tested appropriately that it will perform reliably. And accordingly, they're not going to make it into production. So it's not just because, well, we're worried about scandals, a reputational risk of bad decisions getting made using AI models. It's also the regulatory fines and the cost of regulatory compliance. But at the end of the day, it's also core to driving that impact. So you aren't going to see the innovation and impact that you expect without better improved AI governance. So then question four. So what do we do? What does this look like in practice? And here, it's important to point out the AI governance gap that exists in organizations. A lot of organizations have made progress against creating AI oversight committees, AI governance councils. They've often made progress against adopting ethical AI principles or AI governance principles, maybe even an AI governance framework. The challenges is that maybe they might even have undertaken an AI governance audit or a risk audit. The problem is in the disconnect between all of those very high level activities and what happens on the ground. What is being done as use cases are being developed? How are they being governed? As use cases are being put into deployment, what activities are happening at that level? What activities are undertaken to go and ensure that these models are continuing to perform correctly that the risk profile hasn't changed, that they haven't been adverse events. What activities are undertaken in terms of evaluating new technologies and new off-the-shelf solutions? That's where AI governance actually happens. And if you're not translating from that high level to the ground level, then arguably, you're not doing any AI governance. You're not actually managing risk. You might be going in and providing a legal defense against regulation, maybe. You might be convincing investors and board members and others that you are certainly taking this seriously. But until you are undertaking systematic activities in an AI project lifecycle, arguably, you're not really managing risk. So this is where it's very helpful to go in and look at that lifecycle, to build AI governance from the ground up and look at, so what is happening? What do we need to be doing during the planning stage of projects? What do we need to be doing when we're kicking them off? What do we need to be doing in an ongoing basis? And what do we need to be doing at the end of projects? And before they're going into production, and for continuous improvement. And there's a range of activities. If you are regulated, or if you have use cases and production, you probably already have activities that are happening there. So it's a question of taking those and building on those, going in and ensuring that risk assessments are being undertaken systematically across projects, going in and identifying that there are policies that are appropriate for the risk level of different use cases, going in and controlling access to risky data, risky costly infrastructure, risky tools, or libraries, and going in and having ongoing collection of results and evidence as projects continue. And being able to go and take action off of this, not only just monitoring for performance and bias, fairness, and things like that, but also having processes that go in and take immediate action when it looks like the risk profile of a project has changed or when negative events occur. And then having standard activities that are done towards the end of a project, when everything is going through a final, very comprehensive staging testing and vetting and approval before it's in documentation and tracking all of those assets, that has to happen before something goes into production. So that's really what good looks like. It is looking at those activities across the life cycle and ensuring that there are processes and policies in place that require them to happen. And also that there are capabilities in place that enable all of those activities to be done effectively and efficiently in a scalable fashion that can keep up with the needs of the business. And they can do this also in a timely fashion, so that actions to go in and mitigate risk can actually be taken. That is what we are striving for when it comes to AI governments. So what do we need? What are the capabilities that we need to build around all of this? Certainly visibility. Having that transparency into what the different team members are doing, what the different teams are doing, what's happening across all of these different projects. If we can't see it, if nobody can go in and monitor what is happening and have access to the information that they need in order to make decisions around approvals to go in and assess risk, then we really can't begin. But it's by no means enough. We also need the ability to go in and audit and inspect. We need lineage. A lot of organizations we might have data lineage, but that often disappears the moment we get to the analytics and model development that folks are doing with the data. And so we need that ability to go in and track all of the different important activities that happened from what data sets were used, what code, what versions of the code, what libraries, even what infrastructure, that's something and what environments that things are run on, and the ability to go in and reproduce those. Because if we can't reproduce it, then we can't validate. We can't verify. We can't test it to make sure that it was done correctly. But even the validation and the auditing isn't enough. We also need control. We need the ability to proactively restrict risky activities, most notably when it comes to data access. And this isn't just data access on a role level. This is data access on a project risk-based level. At the beginning of a project when the risks are unknown, maybe you want to be very conservative and only provide access to more sensitive data later on, or potentially vice versa. There are instances where you actually want to do it the other way around. And as you understand the risks more, you actually start continuing, curtailing those risks further. It depends on the nature of the project. But certainly providing control mechanisms when it comes to approval gateways, providing control mechanisms when it comes to, so again, access to other things, the tools should folks be getting access to third party LLMs. And if so, from which providers and so on. Which systems should folks have access to and be able to integrate their solutions with? All of that control and that ability to go in and terminate projects or terminate models that are in deployment where they are behaving erratically or are generating risk. We need those controls as well. And last but not least, we need the capabilities to orchestrate all of this. We need the capabilities to manage policies, to define them, modify them, apply them, and ensure that they're enforced. We need orchestration capabilities around the tasks that need to be undertaken for government, whether that be snapshots, versions, things put into the model registries, lineage captured and tracks, model scorecards created. But also management for the approval tasks. Have the right folks be an approach for validation? Have they undertaken the validation tasks that they need to? Have they signed off on these models going into production? Have they signed off on that they are still behaving as appropriately three months, six months, or even one day later, depending on the needs of the model? So those are the basic questions when it comes to AI governance that all of you should be asking and that all of you should have an opinionated answer to. So hopefully we've equipped you to be asking the right questions and giving answers that will spark the right kind of conversations. If you are interested in more of this to get to more advanced topics, more in-depth as to what are those tasks during the AI ML project lifecycle, or how to assess your AI governance maturity, or maybe even what's the difference between AI governance versus machine learning governance. Let me know in the comments or reach out to me on LinkedIn or at [email protected]. Thank you very much for listening, best of luck on your AI journeys and as always do the right thing for yourself, for your organizations and for society at work. You've been listening to data science leaders from Domino Data Lab. If you found our discussion insightful, be sure to subscribe to Never Miss an Episode. Your support helps us deliver cutting-edge content and keep the data science community informed and inspired. We appreciate every review. Until next time.