Building the Guardrails for AI — with Miranda Bogen

This is Deb Donig with Technically Human, a podcast about ethics and technology, where I ask what it means to be human in the age of tech. Each week I interview industry leaders, thinkers, writers and technologists, and I ask them about how they understand the relationship between humans and the technologies we create. We discuss how we can build a better vision for technology, one that represents the best of our human values. Welcome back for another episode of Technically Human, it's been a beat, actually it's been over a year. And if you're wondering where I've been, well I've been trying to write a book, actually I did write a book. For a while I was trying to write The New York Times Best Seller, and then I was trying to write the definitive book on the history and the future of ethics and tech. It turned out that I couldn't write either of those books because it turns out that what I actually wanted to do was to write a book so that I could start a conversation about the things that I cared about with other people, readers, he cared about those things too. I wanted to create a community for those conversations, the kind that I've been having on this show for years. And so I wrote that book like I was having the kind of conversations I have on this show, and that was the only way that I finished that book. And now that it's done, I'm back for more of those conversations. Today's conversation is with Miranda Bogan, Miranda is the founding director of the Center for Democracy and Technologies AI, Governor Vlad and CBT's Chief Technologist, where she works to develop and promote robust, technically informed solutions for the regulation and governance of AI systems. She most recently guided responsible AI strategy at Meta, leading efforts to measure and mitigate bias in AI-powered products and building out company-wide governance practices. Before that, she was a senior policy analyst at Upturn, where she did foundational research at the intersection of machine learning and civil rights, and she co-chaired the Fairness Transparency and Accountability Working Group at the Partnership on AI. Miranda's research has been widely cited, including work demonstrating the potential for discrimination in personalized advertising systems, and illuminating the role AI clays in hiring. And her technical contributions range from bias and robustness benchmarks to privacy-preserving methods for measuring racial disparities in AI systems. Her work has been featured in the Harvard Business Review, and PR, the Atlantic, wired and even last week tonight. The Center for Democracy and Technology, CBT, is the leading nonpartisan, nonprofit organization fighting to advance civil rights and civil liberties in the digital age. Hi, Miranda. Hi, Dad. So, Miranda, I wanted to start off with your career trajectory, which I find fascinating. You came through political science and Middle Eastern and North African studies approach, and then entered into the arena of tech policy and algorithmic fairness and AI governance, eventually moving through organizations like Upturn, Meta, and now the Center for Democracy and Technology. I wonder if you could talk us through that path, and how it shaped the kind of AI researcher and practitioner that you became? Absolutely. It makes so much sense in retrospect, but I promise I had no idea that it was going to lead to this when I first started. I initially thought I was going to go into Middle Eastern diplomacy. I was really intrigued by intractable problems and how you reconcile difference. And as I was working after college for an organization that facilitated study abroad programs, I was engaged in a lot of digital communications and also digital advertising trying to make those programs known to people who might want to do them. And I was reading a lot of tech news to be like, what are these new features that Facebook is offering or Google for search? And in some of the tech publications, all of a sudden in 2010, 2011, it shifted from just talking about new innovations in the product to geopolitics. That sounded really familiar from my studies. And the one that most stands out was an article with a headline, Google declares Palestinian statehood. And I was like, what does Google know about the Middle East, which is quite complicated. And what did they do that led to that perception? Or did they actually make a statement to that effect and what went into their thought process and do they recognize that they're engaged in geopolitics in a way that states are typically engaged? And how does this play out? And so I went back to grad school and I was really interested in specifically how tech companies make policy decisions, especially when different tech companies were making decisions about the same issue differently. What were the factors that were leading them to kind of navigate those issues, the way that they were? And so that already kind of took my hands on experience working with new technology, engaging in digital advertising, to thinking about corporations and decision making and incentives and power. And I also started doubling an AI at that time. It was still sort of a new concept, but thinking about how my AI shake up geopolitics, there was some conversation around autonomous weapons. And some of the early inklings of frontier AI and what that would mean were kind of bubbling up, but it wasn't the main topic. But just enough so that when I graduated and I went to Upton, I had done something with AI and we at Upton were trying to help the civil rights community understand the impact of emerging technology on traditional civil rights areas. And so with my experience working in digital advertising, this was around the time of the settlements between the National Fair Housing Alliance and ACLU and meta around discriminatory ad targeting. And having run ads in the past, I knew that just because you target people doesn't mean you reach them because everyone's trying to target people and something's going on behind the scenes. And so that practical experience coupled with knowing what the civil rights community cared about, which was people getting access to information about housing or jobs or credit that they could access would be defined not only by how Facebook was designing its product to let people target advertisements, but also how they were using AI machine learning at the time to rank different ads and not really kicked off a research agenda that follows me to this day, not only ads and how AI and ranking systems touch issues that you wouldn't think of as tech policy issues, but also what are the technical research questions that could provide evidence to policy and legal conversations that maybe we intuitively think are true, but you actually need a citation to power some kind of action. And so that's what my co-authors and I did with that research, but also research more recently to say we think something's going on here, we know that advocates are searching for evidence in this space and we're going to go to technical research that we can bring back out that they can lean on in making their arguments around how tech should be shaped in a way that reduces harm to the people who either are using it or who are affected by it. And I think that's as relevant today as it ever was and it's ballooned, the whole field is ballooned and there's so many more people doing that which is great and at the same time figuring out what are those critical questions that need answering and what is the technical research or other types of research that will help us break out of spirals of conversation or really shift the conversation into a more actionable or sort of action-oriented mode rather than just continuing to point out the problems that many people have pointed out. So that's kind of what drives my work now. And I wanted to ask a question about kind of the institutional dimensions of your work because you've worked inside meta, you've worked outside meta, you've worked in the context of industry, you've worked in the context of civil society, including academic contexts. And obviously, or maybe this is not obvious, but to me it's obvious that the incentive structure is behind each of those different approaches yield very different results and very different utility or use cases for each of these results. I wonder if you could share a little bit about what it's like to work inside a tech company and try to address the problems that you're pointing out from inside the institutions that produced it. How does that change from trying to do it outside of the companies? What kinds of barriers or opportunities are available from inside that aren't available outside? How do incentive structures change both what you're able to produce and then how you're able to operationalize or structure the utility of what you produce? Yeah, I went to meta because I had been interested in tech company decision making in graduate schools. So having the opportunity to see it up close was something that felt important to both close the loop and see, you know, was I right? Did I had an intuitive, anything that was going on? But also what could I learn from seeing it at close? And also what could I learn about the technologies that were being built? You know, when I was on the outside, we were really thinking about machine learning and AI, but really looking at academic papers and looking at arm's length. And I wanted to be be proximate with the people building the technology to really understand how the decisions they were making played out in that context and went from an idea to something released in the world. So I like to think about the whole ecosystem as a basket of theories of change. And so civil society is working with researchers doing their own research to highlight issues that are not being attended to to spot gaps in policies and to put pressure on actors of various kinds, policymakers, but also increasingly over the last two decades tech companies by pointing out problems, by raising the alarm about problems, by rallying sort of public attention to problems. And that's really, really important. And ultimately they're trying to convince someone to do something. So you also need the people who will kind of catch that input and figure out how do you reshape it in a way that fits into the operations of the institution that is trying to be changed. And so the way that worked was generally understood, I think, in policy land. Like people kind of know how to advocate for policy change in Congress or in the White House or things like that. It was a skill that many were building to convince tech companies to do different things. And what was really interesting when I moved from civil society to meta was that my role at meta was basically being a policy translator to product, like product teams. So I worked at the AI organization and the AI research part of meta. And basically whenever they wanted to do something, whenever they wanted to touch user data, they had to come through a process. And I was part of that process to say, you can do that as long as you take these safeguards or yeah, we're not going to do that or hear the risks of doing that. There were also lawyers and sort of privacy experts in that process. And they were looking for a signal on how different stakeholders perceive different product launches or details and what would they need to be aware of and what do they need to build into the product to accommodate different requirements or expectations. There was a moment where I was being asked for input on something related to ads, which was not my main area when I was at meta, I was tangential even though I had done a lot of work on it. And I gave some feedback and the product managers responded to say, well, what do the stakeholders that is civil society and an external observers, what do the stakeholders think of this? And I was like, I just got here. I was just a stakeholder like last week. This is what they think of it. And they're like, no, but what do they think of it? And I had to cite myself from work I had done outside, which is just illustrative of the dynamic that is really relevant for people to understand that there are well-meaning people inside these companies who in some cases have formal authority or informal influence and can shape how products are developed. But they often significantly benefit or sometimes even truly need external people to be saying the thing that they're saying inside for it to be taken seriously or for it to be prioritized. And so it's somewhat of a symbiotic relationship. If you actually need both theories of change going on at once, because if you just have the external research and advocacy going on, but no one inside cares or they might care, but they don't really know what to do with that because they're high level principles being asked for or civil rights or human rights impact assessments. If people inside don't know what it means to do that, they're not going to do it very well. They need to know who to ask. They need to know how to operationalize that. And so having people who are embedded who can do that translation makes that work much more effective. But those internal people also really need the kind of cover from the external folks that speak to get the attention and the resources that are necessary to deal with those problems. So living through that dynamic was very instructive to better understand that there's no one way to make change in tech. There are a lot of different entry points and none of them are discrete processes. They're all interacting and being aware of how those dynamics play out will make folks no matter where they sit more effective in doing their work. Yeah, you touched on something that I wanted to pull a thread out a little bit more from. Sometimes in the conversations that I have especially in academia or in, shall we call it, tech critical circles, there seems to be a kind of basic premise that the people working in tech are villains and they're doing things like inserting bias into a system intentionally or without care at all. And my experience living in Silicon Valley of working with a lot of these engineers having them come to my talks and spending time listening to how things actually work internally is that there isn't like this bad engineer or this gremlin working to undo the goods of civil society within tech. It's something I think more insidious. Your work and I think gets at this. It's quite more unsettling that things like discrimination and bad outcomes can emerge structurally from optimization itself with nobody explicitly instructing the system to do things like discriminate or target or draw our attention in ways that can ultimately be harmful. Even when you have places like meta hiring systematically, people with advanced doctorates in how to hijack your attention so as to be able to keep you on the platform longer, the system itself isn't looking at an intentional mechanism for causing you harm. It's coinciding with an incentive structure that is intentionally to the best of its ability seeking to monetize your attention. And I guess I'm trying to get at the kind of structure of institutional incentives and how that works in terms of competing with or overturning sometimes the other stakeholder questions that emerge. Can you talk a little bit about what you saw happening and what academia gets right in its critique of tech and maybe what it gets wrong or misses the mark on? When I was doing my graduate research, I was trying to understand corporate decision making and the main piece of feedback my advisor gave me is that I was treating the institutions as unitary actors and that it was a lot more complicated than that. And I, you know, caveated that in my work, but it really wasn't until I went inside and saw the extent to which these are quite complicated bureaucracies that there are many, many factions inside that are sometimes just tasked with different things and have different incentives that all roll up to general corporate incentives, but even within one team might be tasked with building this type of product and another might be tasked with creating a new line of business or something like that. And sometimes they're actually pitted adversarially against each other where there's like a product, a business team that's tasked with furthering the goals of the business and then their compliance teams who are tasked with reducing risk to the company and they're situated to be adversarial so that there's an internal internally, there's an internally independent-ish actor who's kind of providing oversight into the product development process. Those dynamics then play out and then people are actually adversarial and there's a lot of teams realizing that they're doing similar work and needing to fight it out. Teams realizing that they actually share goals and merging personalities coming into play and all of that is under the umbrella of organizations having their own goals and everyone is incentivized according to their performance structure and the company's success and so there are these micro versions of different goals kind of clashing in ways that sometimes you can overcome by clarifying problems, making problems easier to solve, escalating things and getting decisions from more senior leaders and then some of them hit many, many roadblocks because the intervention to prevent discrimination or to prevent some of their type of harm is at odds with the overall kind of corporate shape of the goal in a way that they're not always necessarily or even many times that I saw necessarily saying here's metric for harm, here's metric for dollar, which one is better, like it's not that explicit, it's more just how problems are framed and how decisions are made cumulatively over time that I think lead to these effects where the public might see a product is suddenly released or harm suddenly happens and say that must have been a decision that was made, it usually was the result of a year or two of product development back and forth and iterating and piloting and launching and and stumbling your way into into that situation not realizing it would lead to that outcome. I think there is more evidence and recent years that there is a purposeful inattention to two problems that for which evidence is very clear and I think that's where just the corporate incentives in general being what corporations are driving for, driving toward is showing up that they don't want to sacrifice the kind of business goals for the level of work that would go into really addressing all of these problems and so they're trying do some amount that is perceived to be sufficient or legitimate while also conducting their business. But ultimately, all business models have incentives, not only the ones that are trying to maximize attention. So my team did a report recently on the business models of Frontier AI and we see some of them that are pursuing advertising as a business model, but some of them are leaning into enterprise software as a service or government contracts or commerce and marketplace type of dynamics and I think an important conclusion of that was all of these business models will shape the products in some way that will have some kind of impact on people's well-being and we just need to pay attention on how those incentives are shaped because they will tell us a lot around what the right intervention points are. So I have a question that touches on some of this and brings in a couple of additional interlocutors. Helen Toner was recently at Berkeley giving a talk on the relationship between American race for AI, so to speak, and China and looking at the regulatory atmospheres in both arenas and also how they interact. And my question for her was that when I what I walk into rooms where I am talking to tech executives or people developing AI from the inside, when I give them the justifications that I have for regulation or policy, we can go back and forth a couple of times and then the inevitable escalation of that conversation is the phrase that if we don't do it and if we don't do it quickly, China will and they'll beat us at it. And what Helen Toner's response to me was when I say how do I respond to that in a meaningful way, she said China actually has very robust regulation and I've thought about that and we were talking before you and I, we're talking before this recording offline and I mentioned that I have been attending the Musk versus Altman case and by the time this episode comes out, that case will have been decided, but one takeaway I have from that case is that the justification for developing AI in any one of these business models to its commodification and market dominance has been if we don't do it China will, but looking at this case and looking at with the kind of underlying logic of the cases, it seems to me that China is a cipher for a different arms race and the arms race isn't between the United States and China, it's between all of these companies and who is going to get the market dominance and therefore the government contracts and therefore not just market dominance, but the kind of autocratic capacity to govern the future of AI overall, including its regulatory capacities and outcomes. I guess, I guess my question is react to that in some way. One of the justifications for if we don't build it China will has traditionally been that we are a human rights respecting country that will advance democratic values and China is not. I'm not sure we're in a strong position of that as we used to be and so the question is what are we building and will it be better and I'm not arguing that necessarily Chinese AI will be better at preserving human rights, but to the extent that that is the argument, we should also be advocating and the companies should also be standing up for the values that make that argument make sense in the first place, but like you said, there are trillions of dollars on the line and from a business perspective, you could see why a CEO runs a business would say it's irresponsible for me not to seek that value because that's what I the business do and some of them have shareholders and could actually be on the hook for fiduciary duty for not doing that and you have to be like really have a strong case as a business executive for why you're not pursuing you know revenue or profit and you can do that if you have a sufficiently long view of you know the life cycle of the company and where risk will come in and what might destroy your value, but right now the upside of what AI seems to offer is so outlandish that it just overshadows the risks that would otherwise maybe give business leaders pause, so one of the pieces my colleagues put out I think it was last summer was an argument that businesses should be more transparent about their risk appetite because it's one thing to do a risk assessment of some kind that's all well and good, but like you can do a risk assessment and then do nothing about the risks because ultimately risk assessments are for businesses to make calls about how much risk they're willing to take on, not a public service that they're doing and unless you have really clear regulation around what you have to act on in a risk assessment, it's not dispositive that you will and so at the very least businesses should be more open that they are just taking on massive amounts of risk both to themselves as a company but also to people in general and I think that's why we're seeing so much backlash in the US against AI whereas there's a lot more optimism in China, I think they trust that there will be rules in place and that things won't go dramatically awry and I think the tech CEOs right now their eyes are bigger than their stomach in a way and they they're just seeing the upside and not seeing all the ways that that the downside will come and not just for people but also for them. There have been a lot of thinkers right now, Tristan Harris comes to mind who are advocating for a kind of human project to create a momentum toward pressing the pause button and every time I hear that I think I think of the curb your enthusiasm episode where they're all gathered around a table eating dinner and a bunch of liquid gets spilled on Larry David and his mother-in-law is sitting at the table and everybody looks shocked as Larry David is drenched in this liquid and she says somebody get a sponge and he says why don't you get a sponge right and so what I hear is Dario Amade saying we need to regulate this and I hear Sam Altman saying we need to regulate this and I see Elon Musk saying we should press the pause button until somebody regulates this and I think to myself all of you are in the position of power to press the pause button more than anybody else in the world why don't you get a sponge so to speak so I guess how do you think about the possibilities for developing something along the lines of this human reclamation project or this ability to press the pause button where do you think that that comes from what theory of change so to speak motivates your thinking in terms of where you think that this can start and where it can go well first I'll say I think the dynamics that have led to this are basically a prisoner of dilemma that like maybe maybe genuinely they all do you think it's be regulated and yet in their position if they stop and no one else does then then then they've lost at the same time a lot of their staff and former staff actually do want regulation and they've told me that and they're like pretty pleased can you help get us rules because I can't do the work I think is important internally when there's not a requirement that's making the organization prioritize that so I think there's multiple layers of real concern real dynamics and performative sort of asks for regulation but in practice again this goes back to organizations have different faces right the the the technical researchers all might really want regulation and these frontier AI companies were largely technical researchers until they were suddenly product companies and so it made sense that for a while they really were leaning into a more cautionary kind of approach now they all have government affairs teams and lobbying teams who are judged on their ability to get to a policy dynamic that's favorable to the company and it's no wonder that they're advocating for for less onerous requirements that's what they're incentivized to do even when internally in the company there might be dissent or large differences of opinion I don't have a particular view on what's the proactive vision that we should be fighting for but I do have a confinary tale that I've been working through which is looking at the history of chemical regulation this was in the 50s 60s 70s where it was clear that certain chemicals led to harmful health outcomes like cancer and no one knew how to measure the likelihood that it would like how much of a chemical how like how much of a chemical would you need to be exposed to in order to be certain or reasonably certain that you would develop cancer in some way and a whole regulatory and measurement apparatus eventually coalesced around this but there were decades where corporate actors like the DuPont Chemical Company were aware of the way that risks could unfold cumulatively over time where that research was not in public and it was not something regulators could act on and even when regulators and the policy community coalesced to act it wasn't like they were taking these products off the market they were trying to figure out what is the limit what is the thresholds that we need to impose that strikes the right balance between the economic utility of these chemicals and the harms that they might cause and I think that's the challenge with AI is that the perceived economic utility so high and the harms are so diffuse that the policy conversation could end up suffering from the same dynamics where It's difficult to consolidate the case for a dramatic reduction or pause or taking off the market when things are already on the market already in use and seeming to provide some kind of upside. So I think there's a lot to learn just from that history. And I was in a conversation earlier today around, well, why are we having so much success with things like data centers where people are rallying and pushing back and having success. And one of the reasons is that they don't exist yet. And there are steps that companies need to go through in order for them to get permission to build those data centers and their intervention points where you can actually stop that process for happening before it happens. Well, one of the challenges of AI is a even very, very, very simple models can cause a lot of harm and so it's hard to figure out where you would draw the line of exactly what are we pausing but setting that aside, even if it was clear. Things are all it's really hard to claw back thing from market in reality. So I hope we can gain momentum around adding significantly more incentive to invest in addressing the harms that are being caused. Those sorts of other harms in society that intersect with those harms. And I imagine there will be structural barriers, especially with the potential upside that like we're fighting an uphill battle. But hopefully we can all kind of come together and and at least fight that battle a little bit more effectively. Yeah, the question of data centers is an interesting one. You point out as your cautionary tails also interesting what came to mind as you were talking was another model, which is the model of bioethics and federal regulations around things like life saving drugs that also merge. And the cautionary tail around there is a little bit different. The cautionary tail in the context of the pharmaceutical industry is around the big crisis happening that is to say the crisis of the Tuskegee experiments and the Nazi medical experiments. We need people to say, hey, as much of a benefit as this science may offer, even in terms of giving us delivering a life saving drug, we need to put some cautionary structures around that because the impulse to say what science delivers or offers is so immense as moral kind of answer to a question as well as a procedure toward progress. That we probably should put some stop gaps toward people being able to justify what they're doing and to monetize it. And so as a result of that, even for something as significant as a technological breakthrough as a life saving vaccine, you have to prove that your product is safe before it enters into the market. So, you know, the caveat there is that the crisis had to happen first before people were willing to say, yeah, okay, maybe putting some restrictions up before we release products and the public is a good idea. But it did happen. Do you see something similar potentially emerging in the context of Big Tech or is the genie service to be gathered bottle as Frankenstein's monster among us already and not willingly going back to any time soon to the lab? I think there will be cases that raise the alarm. I think there already have been there have been very concrete ones and then there have been very diffuse ones. So the first problem is there have been so many examples that I feel like people started tuning out. And so the examples of failure modes and harms are not creating these coalescing moments as much as they did and how do we, how do we elevate the most consequential ones to really motivate action. The second example, I think, is more about a reorientation. The second one is more about a realignment of actors in the space. So with in throttics negotiations with the Department of Defense or the Department of War, there was suddenly a re alignment between the companies and the public interest in a way there hadn't been in a few years because the government was suddenly the one that was purporting or wanting to do something that was really egregious and that was really a rallying moment to say what are the safeguards that we actually want in these technologies in a certain in a certain way. So we saw that it is possible to catalyze that attention around specific events. This leads me to want to ask you a little bit further about people in in charge of putting into play some of those safeguards. The past five years of my research have been focused on what I call the ethical public interest and responsible technology workforce. And I distinguish between those and the taxonomy that I'm proposing because they really do have different job functions, different trainings and different skills. Also, I will put this in parentheses, very different salary requirements and they often belong on different teams. So this speaks a little bit to the organizational structure that you want. They also have often very different labor positions in terms of whether or not they're contract laborers are ongoing wage earners in the context of an employment. And I'll leave you to guess which of the three that is to say the responsible technologists, the people who are working on technical problems or the people who are public interest technologists, the people who are working on regulatory or governance issues or the people who are ethical technologists to say people who are working on asking normative questions or in the most money and have the most job. Security, I think you can probably guess without me outlining it. But the reason that I bring this up is that each of these three different sectors have different capacities in terms of intervening into harms or problems. And one of the things that my research surfaced is that when I started looking at this labor sector or job market in 2021, the ethical technologist that is to say the people who are asking the normative questions, the people who oftentimes had unconventional backgrounds, degrees in near Middle Eastern studies, for example, were about 58% of the overall labor market in this area. Whereas people working in the context of technical approaches were something like 16%. In 2025, the people working on asking normative questions, ethical questions, that had gone from 66% to 8%. And the people who were working on creating technical fixes or technical approaches to the problems had gone up to something like 65%. So it's almost a complete reversal and quite, I think, telling one. Because what we have is the movement from asking those normative questions, questions like, should we build this to begin with? What I call normative questions, really kind of getting sideline, they're oftentimes labor or contract workers, and the people who are asking, how do we operationalize this decision that has already been made into a technical product? That has taken over the field. So I guess my question here is, do you worry about governance becoming performative? Do you worry about the loss of the questions like, should we build this at all? Not just how does this cause harm and how do we mitigate the kinds of harm that it causes? I think the observation of the shape of the positions over time is really interesting. And I'd like to hope that it doesn't necessarily fully change what work is happening. To your point about career stability, job stability, and security, and things like that, I observed that one of the ways that responsible AI teams would kind of protect themselves in times of flux or budget crisis is sort of reorient themselves to appear to be more of a technical organization that was building tools for other teams to take up or moving into more of a compliance space where there was a requirement that would ultimately cost the company not to conform with. And so some of those might be a reflection of strategies in times of constrained resources to shift things around and get that sometimes the people are the same. And so they can bring those skills, but when they shift their orientation to those new roles and those new teams, their frame can also get a lot narrower. So a responsible AI team that's shifting to comply with the UI actor, the DSA, because that's an organizational priority for which their resources available might suddenly find themselves empowered to answer a much narrower set of questions, the ones that the regulations spelled out, because if the organization is perceived to be asking other questions, but under the guise of compliance, then they might be perceived that they think those topics are something they also have to comply with. And then, you know, it creates some complexity internally. And so operationally that can be super limiting for what this work ends up looking like. And so I've been involved in some research that by the time this comes out should be public about the role of fair lending practitioners in banks, which is one of the oldest practices, at least in measuring algorithmic bias in in data models machine learning models, certainly not the oldest practice of of responsible tech and ethics, but one that was a quite concrete and we saw some of these dynamics play out there as well that researchers have have seen in the tech industry more broadly. We thought it would be a. bit different, but there were a lot of parallel experiences where teams that were being asked to bring this thoughtfulness or be aware of a certain type of harm were being constrained in the questions they could ask because of the way compliance was shaped or even though there was a legal mandate to do a thing, they were still needing to appeal using more like emotional terms to senior executives like would you really want your grandma to know that we're doing this, which is I think you might not expect when there's such a clear legal expectation, regulatory expectation that regulated financial institutions are complying with civil rights law, and so I think that just complicates the overall picture that you both need to have the staff who can bring this perspective into an organization. They also need to know how to talk with other people in the organization who might be engineers or product managers or vice presidents. They also need to convince the organization to resource that work and oftentimes those resources are directed to the tech oriented teams and so it just creates a very complicated dynamic internally. I will say there has I think there's a bit of a different dynamic going on if you look just at the frontier AI community because there's a lot of people there who identify themselves as hardcore technologists but who also studied philosophy and are bringing that lens to some of the questions around should we build this or what are the implications of what we're building. It's just that for the most part the implications that they're thinking about are super abstract and long term and don't include a lot of the sociological and societal impacts today that many of the people who maybe would consider themselves like an ethical AI practitioner or technologist or responsible AI practitioner would have been thinking about. I don't know the extent to which the thoughtfulness that they're bringing to the AI their building is sufficient for dealing with the harms that we're seeing like it's good that there's some of that there it's very embedded in the culture that there's like philosophical reflection yet it's framed around a certain set of philosophical questions so I think no matter what position we're talking about what what the positions are called whether they're primarily technical or primarily socio-technical the overall frame that an organization and an epistemic community takes around the problems that they care about I think affects this space quite a bit maybe not influences whether it will succeed or not but it sets the the groundwork for what work is happening at all and then how one would justify intervening if some sort of ethical concern were raised and then you get into a question of what does it take to intervene what are the implications of that intervention what are the competing incentives that might prevent that intervention from being adopted by an organization in whose interests is not clear that intervention would naturally land so I'll have to say I think people can hold multiple skills and can can bring that like ethical attention as well as technical skills but when it over rotates to one or the other it's the work can get stuck because of the challenges communicating across disciplines as well as just the nature of organizations as complicated bureaucracies that prioritize certain types of roles or brothers well I want to talk about those interventions in a second but I could just ask you to expand a little bit on the last part of your sentence there which I think is so interesting you talked about the disciplinary divides and the fact that you know organizations who are hiring different people with different understandings and different vocabularies even in different frameworks have a hard time talking to one another you know when when I for example talk to my students who are mostly technical students and I say the word Python everybody immediately understands what I mean they all have this shared vocabulary for they know exactly where to go and what that terminology means but if I say something like justice or if I say something like fairness or if I say something like harm there really is no shared vocabulary for that and I would imagine that in an organization when you have folks who are there primarily because they have technical training folks who are there primarily because they have a background in policy and folks who are there because they mainly have a background in business are talking to one another those terms not Python but terms like justice or fairness or harm mean very different things and register very differently so how would you go about building that kind of shared vocabulary and understanding is it a matter purely of thinking about org structures and how people work together on teams rather than in silos where they don't have to interact with or confront somebody else's maybe adjacent or even orthogonal logic or does this need to happen kind of upstream of that for example in the context of something like the organization of higher education institutions so that people don't go into the labor market or the workforce with these separate vocabulary set that make it very hard to do that kind of interactive engagement. Don't get me started on the challenge it was to translate the word bias between the legal teams and the machine learning teams at meta one of the things people often scoff at from organizations are institutional values or responsible AI principles or things like that and it's natural to do that and I I react that way often because well what are these principles say about what you're actually doing show me a decision you've made that was different because of these principles but what those types of frameworks and documents can serve as is a canonical understanding for the organization about what values they will make decisions on and also what words mean and that can save so much time realizing that two different teams are talking about the same word in a different way so I think I didn't appreciate until I spent months and months deliberating about the wording of a definition's document so that different teams would know what one another were referring to how important that can be and I think it's organization's mistake to kind of wave that around and say we did a good thing but it actually is quite critical from a practical standpoint so I think the biggest takeaway for people kind of coming into this field or wanting to be effective in it is not that you have to know upfront what all the different definitions are but you at least have to be aware that different people have different definitions for words that you might hear and how to spot when people are using them differently and intervene in that misunderstanding and figure out how to align people around something in particular and add substance that conversation so someone is going to go become a product manager but has some kind of humanities background they might spot that someone's trying to operational as justice in some strange way and could say you know what actually there's a lot more to that can we step back and kind of define what we mean by that and actually interrogate what we're doing toward this goal and that's really really important. My team put out a report last year called Principled Practice which was about operationalizing responsible AI and we talk about some of these building blocks that are actually necessary to have a functional program whether it's called ethical AI or responsible AI, AI safety you need some foundational resources like definitions and terms and priorities you also need an organizational structure that accommodates the fact that there are probably many many different teams doing a whole lot of different things and they are busy their priorities are not yours and you often have to convince them to pay attention to a thing at all at a loan invest time and doing a thing and so we talk about the differences between centralizing responsible AI and ethical AI versus distributing the responsibility for that across teams whether champions embedded in teams or hub and spoke models where there's some kind of center of excellence that embeds people in different teams all of those have different implications for the extent to which a large group of people who are trying to accomplish a whole lot of things all at once can take the expertise that any of these professionals bring and put it into their daily work which they usually have 20 or 25 different things are being told they need to do by tomorrow and even if they would like to think responsibly they don't have a ton of time to do that with what they're being asked to do and this goes back to a question of incentives people are just not in a position to be as thoughtful as they might otherwise like to be because they're being asked to work on a business you know ship like a shipping oriented timeline and so that's a problem across the industry is to get people to slow down enough to even think about what they're doing let alone build interventions that will solve problems because those that delays these timelines and that's what they're judged on at the end of the day and so figuring out how to navigate those dynamics is really important especially for teams that don't have formal authority but have you know ethical weight how do you convince people to do something you have to tell them you have to make concrete the harms that might happen if they fail to you have to make understanding the problem easy so that they can wrap their head around it briefly you have to really spell out the interventions that are likely to be useful so they can take the up quickly and not have to go back to square one themselves when they're not an expert in that thing. And we should expect everyone to be more of this, but these are the practical experiences where people who are just trying to do their job and build their product are being asked to reflect on societal impacts is something that they're like that's I've never thought about it that way. What is it that you want me to do? And so figuring out it translate I think I consistently hear that translation across disciplines and teams is one of the most important skills in AI governance and responsible AI just because of how interdisciplinary it is and how operationally complex the work ends up being in practice. So how do you how do you build that? And I want to go back to something I had asked in my earlier question is this something that employers need to cultivate or that needs to happen on the job? Is this a problem that is downstream from an educational problem where you know both of us are at academic institutions we see that these academic institutions have inherited a much older form of educational modeling where everybody learns their discipline you know you come out as in Middle Eastern studies major or in English major or an engineering major and that's kind of your domain and outside of GAE classes which most students treat as the opportunity to you know play Tetris while they are also simultaneously getting credit for taking your class outside of that they really don't have the access to or the ability to interact with the other ways that people are framing or thinking about problems. So how do you view the solution to this? Because one of the things my research shows is that like you are saying the most important jobs are these what I call boundary spanning roles or translational roles where you're working to translate from one team to another team how do you cultivate that? The interdisciplinary exposure I think is absolutely critical and that can happen by taking a whole variety of classes I think I only appreciated how useful that was when I went to grad school and could take really any class I wanted and it's hard kind of maybe an earlier in education to appreciate why that's important and and why you should pay attention in the classes that are not necessarily in the core area of interest but I also think that professors themselves and like domain specific classes could be more creative in bringing in domains that they intersect within the real world to demonstrate how some of these concepts or circumstances might play out so computer science thinking about a real world product and use case that touches on some kind of history at concept or a political science or something like that and helping people spot that that there are helping people spot that there is connective tissue as a first-order matter and then helping them learn to spot what that connective tissue is and practice drawing connections between disciplines that you might not otherwise put together an interesting thing that I'm observing is there's a common structure in tech companies is there's a product team where there's a product manager and some engineers and some designers and user researchers and they all kind of work together it is somewhat naturally interdisciplinary in that way but with the advent of generative AI and coding tools and things like that I'm actually hearing about more organizations that are collapsing those teams into one person where one person is both the product manager and using five coding to build a product and design a product and that seems like it's losing out on so much of what is already insufficient of the interdisciplinary teams that develop this technology let alone the cross functional partners that as they're often called of policy legal civil rights human rights user research that's that's broader than just that that product team the most valuable instances of where people spotted issues and solved them were when there were when there was cross pollination across those those players because they were able to see things that other people weren't and say oh we're overlooking something and we need to do something about it I was having a conversation with a friend of mine who works at a kind of governance consulting company and one of the observations she shared with me is that increasingly companies are using consulting on structures in order to subsidize the teams that they have cut and this brings up a question for me about internal versus external work and auditing because I think one of the really sophisticated observations in your work is that auditor independence and auditor access is a trade-off between the two the more independent and auditor is from a company the less visibility they may actually have into the system and there's a conversation about this that I should credit people like Deborah Rajee and Timnick Gebruh for opening up with their argument that external auditing loses institutional power it loses the ability to track a product in progress as it goes through the organizational structures and the product life cycle from kind of innovation and inception to conclusion their recommendation is the introduction of what they call the smactor scoping mapping artifact collection testing and and reflection auditing framework but their insistence is really that this work has to be done internally not externally so I'm wondering how especially given what you've just said where you know a person is at the same time the the product manager the coder and the auditor all internally how do you design government structures around systems where we're meaningful oversight requires access to the institutions that institutions are very frequently reluctant to provide resources for are we going to see the wholesale abandonment of any opportunity to do that does the future of this look like consulting or do you have a model that you think might reconcile these these kind of competing factors First I'll say that Deb and Timnick's work was a huge inspiration to to me over the years and including for this work I really think that you actually need all different types of scrutiny during the course of building these systems from very very internal deeply involved practitioners to potentially fully external adversarial scrutiny because they accomplish different things and so I fully agree that there are certain types of auditing and oversight that you need some amount of access for and so thinking about what that internal structure is is really important there are models for this many Fortune 500 companies certainly financial institutions have functions called internal audit or they have what's called the three lines of defense so the first line is typically the business it's building whatever it is they're building the second line is the compliance team or some kind of internal oversight that is at arms length has some independence they're not incentivized for the same thing as the as the business team and then even a third line that's internal audit checking that that all that's done that's a pretty well established structure and the internal organization accommodates that independence internally so there is some arms length analysis going on it can spot things but the role of that team is to surface risks for the business again to then decide what to do with the knowledge of the risks that it's taking on and businesses can choose to take on risk and so that's where external auditing might be really important where if you need to publicize or at least highlight some entity that can have some kind of power or enforcement that there was some gap in practice or some harms were happening it might need to be an external party to do that because the internal folks are not in a position to do so I think we are seeing actually a demand for more external auditing by companies by employers of AI who want some kind of independent analysis of products that they're buying and they don't trust the developers to be telling them the full picture or the developers just haven't done the analysis that's relevant for the deployers so there's a community standing up of algorithmic auditors AI red teaming organizations folks who are doing external analysis of AI systems and there's an understanding of why they need to exist and they still face a lot of these problems of accessing the information they need to do their work and so I think all of them play different roles all of them can engage in different work and figuring out what is the goal of a given oversight activity will help identify where in that spectrum one would need to fall and in our report where we look at this analysis of auditing we tried to articulate every single theory of change and goal that an auditing activity could have and place it in the universe of the types of auditing that might bring that about just help people spot where one might be effective and where it might fall short. I want to ask a particular question about AI and the ways that I think even well-intentioned attempts to do things like mitigate bias or reduce risk or promote safety can go adrift. I know you have some recent work on on fine-tuning and what's called safety drift that raises a very profound governance problem, because what you found is that even routine, well-intentioned fine-tuning, can unpredictably erode safety guardrails and that the scale of tuning didn't actually reliably predict anything about the severity of the failure. Can you walk us through what you found and what it means for how we think about responsibility when downstream systems cause harm as they do? Yeah, the current policy conversation is really grappling with who should be responsible when something goes wrong with an AI system and developers are pointing at employers and employers are pointing at developers and none of them want to be on the hook. But there's some co-lessing around a concept of substantial changes to a system. Like if a system was substantially changed from the time it was built to the time it was deployed, the entity that changed it probably has some role to play. And we wanted to better understand what constitutes a substantial change. There is an assumption that like the amount of a model that changed would indicate whether the change was substantial, but we hadn't seen evidence of that effect. So basically we took a lot of different open-weight AI models and we fine-tuned them using various methods for benign purposes. So getting better at medical contacts and getting better at providing legal advice and doing those types of fine-tuning would be harm reducing interventions in many cases. If you were deploying an AI model for a medical context, it should be good at medical things. But what we found was when we tried to make a model better at like understanding medical information, it suddenly would offer very strange advice, even medical advice, and also break in all sorts of ways that were related to medical tasks. And so it just raised the question overall of what types of changes will actually lead to substantial shifts in the safety characteristics of a model. And the answer is we don't know, it was entirely unpredictable how large a change was, whether safety got better, safety got worse, whether two measurements of the same notion of safety moved in opposite directions, which was quite concerning. But I think it's indicative of a broader challenge in this space where people get excited or invest a lot in interventions that seem like they're helping address the problem. And ultimately that doesn't end up solving the problem where it creates new problems. So even going back to something much simpler like algorithmic fairness, there were many cases where I saw teams, you know, measure outcome disparities across different populations, and then be like, okay, cool, how do I twiddle the dial so that the numbers are better? And they're like, no, no, no, no, no, that's just papering over the problem. And probably won't, even if you are really well-meaning and are trying to advance equity, if you just are trying to make the numbers look better, you're not actually addressing the root cause of what led to these disparities in the first place. And so what you really need to do is use the measurements as a barometer and a guide for where you should be looking to see what maybe went wrong and to do a root cause analysis and try and actually solve the underlying problem that led to those errors and fix that and then measure it again and see what happened. But that's actually a challenging ask for people who are in a very engineering mindset who are like, you've given me a metric and now I will optimize that metric. Why is that not enough? And so that's really where again, some of the cultural differences across disciplines come in that saying like just because we've spotted a problem in this way doesn't mean using the same method that you use to measure it means is the right way to mitigate that or just because an intervention seems logical as a way to deal with this bug or this harm that's happening on this platform doesn't mean that's the right intervention given how this harm plays out in the broader world. And that's one of the challenges with tech harms and AI harms is that we can see the systems really well and we can measure these systems sort of but these systems are interacting with the world and a lot of the information around how the harms manifest and what support people actually needing navigating them might be offline and that's sometimes an unsatisfying reality that some of the interventions are maybe better suited offline than online but the online you know entities the ones who are building the technology have the most resources and we don't want them to be off the hook for something that they're exacerbating but trying to solve the problem only in the silo where the technology is doesn't necessarily solve the underlying problem that was ultimately being amplified and so figuring out how you put these puzzle pieces together and how you incentivize action across the right actors and and you know don't let the tech developers off the hook for things that they could do more on but also recognize the systemic interactions of harms in ways that that do take a lot of thought and care and interdisciplinary collaboration and time that's really important and what's like the real challenge is we don't have the time we could have the time but the incentives the economic pressures mean that the time that we used to have to grapple with new scientific and ethical questions has collapsed to a pinpoint where we used to have several years and I think that's one of the biggest challenges right now and so the efforts to kind of pause or slow down to the extent we can buy ourselves time we'll be able to do more whether we'll be able to do enough I think is still a question but if we can get more people focused on using the time we have to spot problems and recommend and convince people to adopt interventions will be at least in a better place than we were before I admit I am an incrementalist and that is a theory of change that not everyone shares and that's okay and I think that the calls for more abolitionist and more black and white solutions or sort of a particular line in the sand solutions can create space for incremental work as well and so that's actually super critical sometimes there's a critique that incremental approaches undermine those types of interventions and I appreciate that critique as well and I think we just all have to kind of grapple with what is the right theory of change given the political and technical landscape of a given era and when would we change our mind and what do we do with the fact that different people work better in different theories of change I think we have time for one more quick question and I will really make it quick one of the observations I have from from your last answer to my question is that it's much easier to ask questions that have measurable outcomes or measurable answers and it's a lot easier to ask questions like how do we mitigate this harm it's a lot much more difficult to ask the more broad philosophical questions such as the question of you know whether or not we should be simply measuring our movement in AI based on whether or not whether we are mitigating harm I wonder whether harm reduction alone is just too narrow of a framework technologies obviously don't just reduce or create harms they they don't just work with that fine-tuning knob they shape relationships and institutions and labor structures in ways of understanding one another and ideally they don't just cause minimal harm they they make life better they cause human flourishing that's I think why we pursue technologies or why those of us who adopt them adopt them do you think that AI governance conversations are still too reactive and when we focus on preventing harms what other questions are we missing about what features we want not just the ones that we don't want are we spending enough time asking the kind those kinds of questions the kinds of questions about the societies that these systems are actively producing whether or not we want those outcomes I do think AI governance can be very reactive and I also think I think it's necessary because without it there's very little of that due diligence going on but it's certainly insufficient for the broader project I think people can also get preoccupied with their slice of the intervention kind of field because that's our nature to like do our jobs and I do think figuring out how we have those broader conversations but in a way that will change the incentives of the actors at play is really critical ultimately there are a lot of powerful forces that are shaping the development of this technology at the moment there's more attention on it than seems like any other technology has had in at least some time and on the harms it could cause all at the same time and so trying to parse that out to figure out what is a genuine benefit that we could derive from the technology and what is marketing fluff that's being used to justify in investment in a technology that hasn't sufficiently gone through some reflection on the impacts it will cause is genuinely challenging at the moment I think we're hearing a lot of ground soul from people that the future that the tech companies are building is not the features that they want and I think the question is how does that translate into actual constraints and I think in a democracy usually the answer is the lawmakers reflect the will of the people and create some rules and actually try and shape behavior within those and right now that process is is broken and has been broken and I hope that lawmakers will be motivated enough to realize that they have to act because people are already seeing what could happen and they don't want that and and yet the again the sort of bigger economic races are so powerful lawmakers I think are scared of you know poking a system and having having consequences that they didn't anticipate just like when you change your model things can change in ways you don't anticipate the economy is like that global governance is like that no one wants to poke it in a way that could lead to an economic collapse so they I think they're trying to weigh people's lived experiences on a day-to-day with the long-term economic health of their country who whichever country we're talking about and and do so in a way that doesn't cause too much harm all at once and like after you know a decade or more in policy I think that it's just a very unsatisfying reality of like it's all trade-offs it's all choices none of them are perfect and everyone's kind of muddling through and you need both the tinkers and the risk managers and the visionaries and the activists to kind of soften the corners while also working towards a future that is better and those are different types of work and they are both important and one can't I think solve all the problems without the other we could be working for a visionary future for decades and what do we do now with the systems that are being deployed and you know we could probably reduce the impact the negative impacts that have just reducing the negative impacts will lead us to the future that in several decades we hope we get to so I hope that people kind of see where they want to sit in that ecosystem but not begrudge other people for choosing a different approach to advancing good and reducing harm because I think we need as many people as possible Miranda thank you very much thank you this is great [Music] [BLANK_AUDIO]