AI, Accountability, and Civilian Harm

[Music] Welcome to this Sick Them Final episode of our mini-series exploring how states respond to Spillian harm from their own military operations. I'm your host, Mae Thompson from Ceasefire, the Centre for Civilian Rights, and in each episode we'll be joined by leading lawyers, experts, practitioners, and military personnel to discuss the evolving landscape of military accountability, the gaps in existing redress mechanism, and the potential for UK's civilian harm from redress scheme. This mini-series is part of a joint project by Queen's University Belfast, University College London and Ceasefire, aimed at advancing legislative amendments that will provide reparations specifically in harm by the UK's involvement in welfare. The project is funded by the UK Research and Innovation Arts and Humanities Research Council. I'm delighted to introduce today's guests, Luke Moffett, Chair of Human Rights and International Humanitarian Law at Queen's University Belfast, and leading expert on reparations for Spillian harm. He is the author of algorithms of war, the human cost of AI in conflict, forthcoming from Bristol University Press. Jessica Jorsi, Assistant Professor of International Law at Utrecht University School of Law, whose research focuses on the legitimacy of military targeting operations in light of increasing autonomy in warfare. She is the Director of the Realities of Algorithmic Welfare Research Platform, an expert member of the Global Commission on Responsible AI in the Military Domain, and ambassador for the Lawful Bydesign Initiative and an Executive Board Member of the Civilian Harm Monitoring Organization Air Wars. And Chris Rogers, Senior Fellow at the Recentre on Law and Security at New York University School of Law. He recently served as Branch Chief and Law and Policy Advisor at the US Department of Defences Civilian Protection Centre of Excellence, where he helped lead efforts to strengthen Spillian harm mitigation across US military operations worldwide, advising on Spillian protection, civilian harm investigations, engagement with US allies and partners, and integration of emerging technologies, including artificial intelligence. Thank you all for joining. You have such a wealth of expertise between you, so I can't wait to get into this conversation. We usually record these episodes online, so actually it feels really special to be in the Lawful Studio with you today. And for today's episode, we're talking all things AI, which has really been dominating the discourse across the Defence and Security sector, that something that feels somewhat overlooked is the potential of AI in civilian harm mitigation using it in a positive way, and how it could be used to track harm and support the redress process for harm civilians. So Jessica, perhaps you could start us off, just by setting the scene for us, by describing how AI is already shaping the way that militaries plan and conduct their operations. Yeah, sure, and thanks for having me, it's exciting to get to talk about this stuff with you all. What we're seeing, so military AI is of course very broads, we're seeing it across the entire spectrum of military operational planning preparation, et cetera. What I look at mostly is what I term the sharp end of the stick, where where militaries are integrating this technology within the joint targeting cycle, for example, and they're integrating AI across this cycle, collecting and analyzing data, these systems help with identifying patterns. They can also feed into recommending and nominating targets. It's marketed as decision support oftentimes, but in reality, it's really starting to shape how decisions are made on the ground, so it's really feeding into cognitive ways of thinking about war. And I think from where I'm sitting, the risk that I see is that these systems are quietly displacing human judgment. They're sort of creeping into places and spaces. They're meant to assist us in thinking about how to do our jobs better, outside of the military context, but across all sectors. I think here though, we're seeing the effects of biases creeping in so operators can fall into automation bias, differing very much to the output of these systems, or anchoring bias when a system nominates a target that anchors, that's a heuristic psychologically for us. That's our starting point in that sense, that anchoring bias then makes it difficult to shift away from that. We're also seeing indications from some ongoing conflicts now of over trusting machine outputs, losing critical decision-making skills, so the idea of cognitive offloading, and then the de-skilling that may occur because of that. We've seen this in other sectors, and I think that's important to understand that AI is interacting, and we interact with AI in all kinds of different ways, but it's not specific to the military sector. So what we're learning, what we're seeing from, for example, how doctors are integrating these technologies in the medical sector, we can also learn from the kinds of effects that we're seeing there. So recently there was a study done in the medical science field, where doctors, oncologists who were decades-long, decades-long experience of detecting cancerous cells in their patients, they started to farm out some of that to these AI-enabled decision support systems, and then there were, there's empirical evidence now that that is how to de-skilling effect for those doctors. When they came back after having used these systems for a while, they were less able to detect cancer, right? So understanding what we can learn from those sectors in the military context is helpful. I think when we see the last point I want to, you know, raise to set the scene, when algorithms are influencing key choices, accountability and legitimacy of operations can get blurred. And I think that militaries, it's vital that they keep context-appropriate human judgment and control at the center here, ensuring that the machines are, and the systems are informing, but not replacing that human responsibility. And the work that you're doing just to be clear is not a hypothetical plan for future use. This is something that militaries are already doing right now. Absolutely. Yesterday, today and tomorrow, and I think that if there's one thing the listeners can take away is that please stop futureizing this. It is the future of warfare, but it's also the now of warfare. This is happening. Now it's something that most militaries are interested in or already developing, acquiring, procuring, etc., and integrating into their own arsenal. Yeah. And what we're seeing is these emerging technologies are, from our view, it cease fire, posing heightened risk to civilians. And Chris, we've seen in recent years that civilian casualties are skyrocketing, and that seems to have correlated with the military use of AI. Perhaps you could just start off by explaining to us what forms of civilian hammer associated with the use of military AI, and do you believe that there is this correlation between the scale of civilian harm and the rise of autonomy? I think that there's certainly, as you say, correlated. I think it's difficult where we're at right now to draw a straight line, certainly firstly between the introduction of autonomous weapons, which I think is, we're at the really the very, very earliest of stages of that, which to a degree is a bit different than I think what Jessica was alluding to, which is the introduction of AI, decision support systems, the integration of AI into things like the joint targeting cycle, which I think we're seeing too much greater extent. And I think that that is contributing in certain contacts to higher levels of civilian harm. I think again, we're in early stages of this, so it's difficult to link so directly or to draw that causation so conclusively, but that's why it's important to draw more attention to this, to invest more in studying this and understanding how these systems are being deployed and integrated by militaries. What systems are in place to ensure that they're done in ways that are supportive of law and IHL and civilian protection and not eroding that. I think Jessica went through a lot of the various ways in which we've already seen these risks being manifest in operations and we've seen a through line between those risks and the types of harm we've seen, for instance, in Israel's operations in Gaza. And that's both, I think, related to the scale and scope of the operations and the ways in which the types of targeting and identification of targets that AI has enabled and perhaps the erosion and uncertainty around how AI is determining those targets is contributing greatly to civilian harm, likely a significant number of civilians being targeted, despite not being combatants and violation of IHL. And then just in virtue of the, again, the scale and speed that AI enables in terms of targeting and the kind of automation bias and the kind of perception of reliability over reliance on the kind of target nomination identification by AI. I think also feeds into, again, the level of and the scale of of air strikes and an attacks being conducted, which again, you know, leads to not just civilian harm in those strikes themselves, but also has cumulative and reverberating effects, which I think absent the use of AI and the escalation of the scale and scope of those operations, supported by AI, I'm not sure we would see the same type of levels of cumulative and reverberating harm that we've seen. So then on the flip side, when it comes to civilian harm mitigation and how algorithms can be used to support that, I'm really interested to hear from your time at the US Department of Defense, Civilian Protection Center of Excellence. If you can, what you are able to tell us about how the US was starting to think about how it could be used. Yeah, I mean, I think, you know, well, perhaps first and foremost, I think it was important that the question was being asked and it was being asked from within a military and within the DOD in particular. And I think without those questions being asked within militaries themselves and then investing the resources, the capital, the capital. the attention, the institutional energy into that side of the ledger, it won't happen. So I think that's one takeaway I had from the time and the discussions that we had within DOD. And I think we started to explore, again, areas where we thought potentially the development, further development and research with respect to different AI capabilities could support mitigation, assessment, etc. A couple of those areas, for instance, the combining and integrating of many different forms and flows of data, particularly open source information with military intelligence, ISR intelligence surveillance, reconnaissance information, and being able to integrate triage, correlate, corroborate all those flows of information in much more real time and dynamic ways. That we thought we think could yield some real gains in terms of enhancing the assessment, civilian harm assessment process, and possibly in circumstances such as in large scale combat operations, where the potential scale of harm, the allegations of harm to sort through are much higher, some of those capabilities might also help equip militaries to better handle those circumstances and to respond and assess allegations of civilian harm at a much greater scale. And Luke, I wanted to ask you about accountability. I'm going off script slightly because I was going to ask you about the implications on accountability when AI is used by militaries, the impact that has on civilians, but also if you could speak a little bit to that, but also I think what I really want to ask you is about the impact on accountability when some of the things Chris has just been talking about, in terms of supporting data collection and tracking civilian harm, how could that support the accountability process for civilians that have been harmed? Yeah, I think there's a wealth of information there that could really help to ease the burden on civilians to show that they were harmed from an incident. The difficulty is, and it's a broader debate within the lethal Thomas weapons systems. The ongoing negotiations is we talk about AI being this black box. You need some people to talk about being a double black box. It's actually a triple black box because algorithms themselves, especially when you use machine learning, are really hard to penetrate and understands how the different weights and thresholds got to that output based on the data we choose. The second sort of box that that is in is my military sick, and in the third box, like almost like Russian dolls boxes, but it's who's accountable for then using that data that was that output that came from these different types of algorithms could be a number of them. And that this triple black box is particularly difficult for civilians because if you think of that case, which are being brought at the ICC, such as Ryan Gaza, where in the first 27 days, the ultimate bombardment there are 12,000 bombs were dropped. I will just talk as decided. We've had lakes and IDF officers coming out and speaking to you, Valia Brim and the 972 magazine indicating that these were using ICC system, so it was data being churned up and given outputs, which human operators then give the green light and kill chain operations, we were really, you know, talk about this rubber stomp and that they had maybe 20 seconds or decide and be a send gender, whether somebody was a target. And that's what you know just was said about in terms of, you know, this automation bias was anchor in this, that humans lose their cognitive ability to first comply with the law, but also to be a human being that when they're based on that decision on what a computer is said, it's going to have a result of causing harm to our individual. Now under the laws of war, it's permissible if that person is combatant or is somebody's directly protesting us all these, but the danger is then it just becomes a way of green light and a lot of killing. And I think it's going to be very difficult to penetrate and hold their kind what actually went wrong because you've got the developments, you know, the program and secrets of how the algorithm is developed in the first place, the, the, you know, nice security issues of rhymes, the proportionality calculus about how those targets were selected, like a whole range of issues that's a village you never going to know and the public doesn't know. And this is very problematic because even if we can't hold individuals that kind, we can't put, you know, algorithms on trial, this state is ultimate responsible for buying and procuring and developing these systems. And then use them on the battlefield for civilians and for even taxpayers, we have to hold our governments to account and the B transplants, so there isn't this opacity in these black boxes around high war is weage that there is some sort of restraint. And so I think like I was at a talk a few months ago, I'm in Liverpool and one guy said that, you know, artificial intelligence doesn't exist. And so what we're doing is, you know, educated guesses based on past data and perhaps a campaign dynamic where we've got a fusion of new data that's coming in, but really we have to be more cautious in terms of how these things are done. And we've seen this in the past with precision bombs were war and a lot of people talk about this being sanitized by how we're able to use technology to hit the enemy and kill the enemy. But what was happening most in Raqqa Gaza is that precision weapons perhaps were being used, but dropping a 2000 precision bomb on a target can hit that target within 5 to 10 meters. It doesn't limit the effects. And so, you know, as you're talking about, there are very written effects of those bombs. Like it's, I think, an aligopile talks about with Mosul. It was deaths by a thousand precision strikes and so that that constant bombardment law hasn't really thought about that changes from being individual precision strikes to the cumulative harm exercise. Or really, it's just we're back to the second world war and carbon ball in our enemies and not recurrent about these other human beings are caught up in this massive wave of violence that is completely removed from our lives and are concerned. So, like, I think it both, you know, as Jessica was talking about that it changes my people think about war here using the weapons, but also for us in countries which are procuring, which a few miles down the road they're developing these weapons and systems is that it's desensitized to the effects that we're seeing the potentials and we're seeing how we can win more words and defeat our enemies, but we're not able to penetrate and the whole to kind our state for the massive harm that will allow us to be able to do that. And the massive harm that will likely result. Can I come in on that because I think that's such an important point. You said what, what does the law have to say about this? And I think that's very interesting to bring it up and I think Chris, you also highlighted the difference between autonomous weapons systems. The attention that discussion has had for the last ten years, there have been really structured international discussions about this. The CCW convention on certain conventional weapons, the group of governmental experts on lethal autonomous weapons systems and I've sat in on those meetings and it's actually really constructive dialogue. We're not there yet, but these things take time diplomacy takes time and deliberation, but all of the oxygen in the room has been taken up by that particular very niche element of autonomy on the battlefield. Whereas the AI-enabled decision support systems, what we're all talking about right now, have gone unregulated. And this is something that I think we're starting to see change, but I just want to take this opportunity to highlight the of urgent need for more regulatory focus on this. Because without guardrails, this will become the norm. I'm afraid even if states and I do genuinely believe states want to be the responsible users. But if we don't give the guardrails, if we don't give meaning to what that legal framework looks like and what kind of cognitive effects these things are having on the way decisions are made in warfare, then I think we're doing a disservice to our militaries, to our industry who's trying to design these things. We've got to put more focus on regulatory efforts in this space. And thank you so much for clarifying that because that's someone who's not so deeply entrenched in this specific area of work. I think that was my bad actually for conflating algorithmic and autonomous and AI. No, but I think it's a really important point from maybe more and outside of perspective. It does feel as though definitely the autonomous weapons issue is receiving a lot of the attention. And that makes it easy for us to when we're thinking about these things that only be thinking about, you know, the autonomous weapons systems, but your right is all of the decision support systems that is, yeah, aren't maybe having those guardrails because they're not being seen as potentially as lethal. But thank you for raising that. And I mean, this is to all of you and it comes to the point of it's not only militaries that are using these things. And when it comes to sort of supporting civilian harm mitigation and tracking investigations response, there's also use of AI and algorithms by NGOs and non governmental organizations by civilians documenting the harm themselves. So I'm not sure what my question is, but I think did they also need to be thinking themselves about what their limits are and how they should be using these systems. You know, I feel like at the moment it's just sort of a free fall and states are able to wait until there is some sort of gunning legislation around this. But for other actors, what perhaps would you say the necessary next steps? I'd say something very, very general. And there's a great series that was recorded in the 80s, like 89 of about a Canadian-chemist physicist at Ursula Franklin. and she talks about technology and high technology. as change the way we lived in the 80s. But she said some really important things, because she was going to hide the Chinese thousands of years ago, create these ornate copper structures. And she was talking that, with technology, it's prescriptive. In other words, like algorithms, it's step-by-step, and it gets you a certain process. But it's a culture of compliance, and it gets into the fixed mindset of how a problem should be done. So I think with NGOs, common to the C-Shift-High, we can use AI. This machine learning algorithms is there's clear de-indition risks. We know a lot about the bias, and how the data sets are done. But also, the loss of cognitive function. We've been talking a lot about the last couple of days about this. But even just going on the grind, an interview in people who are affected by it, and not just in a data collection that does involve people affected. So it's a more people-centered approach. That tells us a certain story, but it isn't a whole thing. And I'll leave in this point here, but there's a famous computer scientist, a feminist theorist, who talks about Catherine Hillens. Who talks about human machine assemblages stuff. But she says, "We make a toast, and the toast is made of gas." And so there is, the interview with High, these two are wielded, that it has an impact on how it's done and used, and we need to be careful and think how we're going to leverage the benefits. But we also need to think about how that then shapes and changes us. I'm just concerned that AI is a high bubble, and it can't do many things, but it can't do everything. It doesn't replace humans whatsoever, and it isn't that intelligence. It's very vulnerable to poison and whole range of issues. And the more we lean and rely on it, the more then it weakens ourselves to stand up and do these things, and report and document and advocate for civilian harm. So, we're the caution. It is to come in on that completely agree. And I guess I would say also, you know, many, much of the work that I think many of us have been involved in, or organizations as well, at some level, is trying to shrink the distance between those who are using force and wielding military power, and those who are affected by it. And modern war, precision-guided munitions, autonomous systems, you know, these things tend to elongate to distance even further that relationship, which I think is morally problematic, leads to more harm, legally problematic. And I think one of the ambitions, I think, of many civil society and NGOs and organizations that work in the space, again, is to try to create more connectivity and shrink the space between who is using that force and who is being impacted by it. And so, to that point, exactly because making, I think it's integral that organizations that are thinking about integrating AI-enabled technologies into their practice and documentation, also retain and sustain that relationship to the communities and the civilians who they're trying to engage with and support and empower and engage in that documentation. I think, you know, another reason also is someone who's engaged in that type of interviewing and documentation is because a lot of times it's through that documentation interviewing that you understand and problems and challenges are revealed by civilians themselves and talking within their context and things that would otherwise be invisible to other forms of data connection and analysis, right? And that's not to say that that type of interviewing and that type of documentation can't be helpfully complemented by those technologies, right? But it's not a substitute by any means for that. And if we were to pursue or fall prey to that type of techno-solutionism, I guess I would say within the NGO and documentation community, I think many pitfalls in that and at core, I think it would be to move away from that goal of trying to shrink the space between those who are impacted by conflict in those who are wielding the power and the military force. And when it comes to positive uses, I'm just interested in your work and your advocacy, whether it's almost the argument of being able to complement sort of more traditional forms of tracking investigation with AI, whether that just detracts from your advocacy for safeguards and guardrails around military uses by saying, just paying devil's advocate if you're approaching a state and saying, "Oh, well, you need to be really careful about how you're using this or we need to be considering all of the possible risks." But we've identified that this is a positive use. So, okay, like Greenlight, go ahead and do that. Is that ever something to consider or how well do we need to balance how much we're promoting its positive use? I think that's a great question, but I think it's a technology here that's agnostic, right? It doesn't really have different. Good, no good or bad. Not necessarily, I think, in what was interesting in these discussions we're having here in Belfast with just such a rich expertise of human rights investigators, et cetera. If I closed my eyes during some of the discussion about these positive uses today, hearing about why human rights organizations and tracking organizations are interested in using AI, it's for the exact same reasons that militaries are, right? And if you extract it outside of this discussion, it's the same reason any sector is interested in pursuing AI-enabled technologies for efficiency sake to speed things up, to take away the dull, dirty, dangerous stuff, right, to allow humans to flourish. But the same concerns that I have for militaries that I've just outlined about the biases, et cetera, the cognitive shifting, those are equally applicable to those of us doing advocacy work or engaging with militaries, right? So I think, again, in a perspective of agnosticism, these issues arise in the medical community, in the military community, in any community, like those of us teaching classes, we're concerned about our students, are they able to critically analyze texts anymore if they're just throwing them through a chat GPT to get a summary? But that same exact concern that I have about that, I take to discussions at NATO when they're using a maven smart system to do the same thing, to summarize a 200-page document because it can do it much faster. It's the same thing in that sense. So I think the positivity, and back to an earlier point you brought up, the difference between autonomous weapon systems and AI DSS, I think there's actually a lot of lessons to be learned. The 10 years of discussions on AWS are really valuable also for particular elements of DSS as well. So we need to learn and talk across these boundaries in order to identify the lessons that we have to learn and then consider that as we start integrating and using this technology. I pretty much agree, and I also think organizations should be propositional about what the responsible integration of these technologies should look like. And I think A way to be propositional and to be credibly propositional is to understand the technologies well, to be fluent in them, to understand what they do, what they don't do, and in some ways to integrate them into one's own work, but to do that responsibly. And then wrestle with what it means to do that responsibly. And then when you wrestled with that, even internally in your own work and with others, I think you're in a stronger position and a stronger posture to be propositional about what that should look like in other spaces, including in the military. And then to ask the hard questions from an expert place and from a knowledgeable place and to push and advocate for what shape that should take. And I find this word responsible really interesting. The UK's policy approach to military use of AI, the title of it is, I think, ambitious, safe, responsible, and they have a lot of emphasis on the fact that even though there are maybe some of their adversaries that have been seen to be using these systems in a unresponsible way, that they would be the responsible user. And I think I'm always skeptical of states saying that we don't need any legal boundaries because we'll be the responsible user. We don't need to worry about how we would use the systems. But I do think that that does. It is alarming to me that we would put this trust in states of being responsible. So Jess, perhaps, is a question for you around what do you think the role of the law is in this case? What extent do we need legal boundaries? And then are we relying on the actors to discern their own responsibility? Yeah, I think that as a lawyer, there is a certain level. For my poor arm sitting anyways, the way that I look at law and responsibility, like to understand what a responsible user is, to me, that means you better be compliant with your legal obligations. Right? So is that what we're talking about when you say you're going to be a responsible user? And I think then when states claim that, I think we need to take a posture of trust but verify. I want to believe and I do believe you're trying to be responsible. Well, what does that mean? So then start peeling the layers back of what that means. And that's where an initiative like the lawful by design approach can come in handy. There's all kinds of these new by design things popping up, ethical by design, responsible by design, lawful by design. But the idea is basically taking a really granular approach to what are these systems And I absolutely echo Chris's point about having to move out of your own disciplinary lane and dip your toes into understanding the technical background of what these systems are and do. You don't have to become a computer programmer yourself to understand what these systems are, but experimenting with them yourself gives you some knowledge, some foothold to start engaging with actors and engaging in a way because I think law is not an afterthought and we have to push back on the idea that these off-the-shelf systems, that's the first time we should, once we procure them as a state, that's the first time we should be asking questions. We try to flip the script on that with this lawful by design and think about it from the very earliest stages to build law in rather than have it be bolted on because it's just not possible, right? And I think we work to bridge that gap in a way between industry innovation and legal review processes to really ensure that the law is baked in in the sense and that not just law but also ethical obligations and compliance at what point are industry representatives thinking about it, if they're not yet thinking about it, I encourage them to think about it, they should be thinking about it and I think it's in their interest as well. When they're building a product, if it's able to be compliant with IHL, then states are able to easily integrate that into their systems, right? That's a easy thing rather than all the R&D and all the investment that goes into building a product, if it's not something that's going to be able to be legally compliant or ensure that states can be legally compliant with its use, there's no sense in actually building a business case out for that. And I think that life cycle approach is talking about all of the stages of the AI life cycle from design, through testing evaluation, verification, validation, deployment and all the way through decommissioning and right and it's a cyclical approach but it's also it's not linear in that sense. Like you have to sometimes go back to the drawing board at places you didn't envision, but this is a challenge and I think it requires this multi-stakeholder approach and being comfortable asking questions of other disciplines and other expertise areas to learn what you don't know so that you can have that credible conversation that Chris was mentioned. Absolutely and just to sort of bring in our project Luke around a UK statutory scheme for civilian harm, redress. Are there any areas specifically where you see that this would be beneficial? I'm thinking of redress in terms of compensation and amends but also the mitigation aspect of redress as well. I think like we talk about you know man's condones payments, redress and I think fundamentally what all those things are and the reparations as well is a way of acknowledging the harm has been caused and it's very difficult for those who are using violence to do that because they believe they were compliant with their Muslim engagement with the law and that are things to happen. It's just it's incidental so it was an intentional. It was awful but lawful that Chris Lager talks about and I think when it comes to the problem we always feel the reparations is the difficulty of victims bringing claims in a timely way. So in human rights law we talk about reparations being adequate, effective and prompt and they never are prompt at least because time is the enemy here that's victims if they're injured and as family members dead, a breadwinner in particular, the family gets worse off, it becomes more impoverished, gets health complications, people can get human trafficked, you know there's a whole range of cascading arms and if you think about that in large of communal societal level it's going to make things worse off. So I think there's space for high algorithms could help verify and support victims' claims. We've been talking about the last couple days with some options around this but really I think it's a point Chris did to you me that there is a word to focus out with a lot of this and it's Jessica Sandisharp end of the stick that it is about the thaliday, it's not about how do we go about medicating the harm. I think where I all got into this sort of area was working with victims that for years, competing for a redress and what's happened is trying to get up wherever of not being in the situation where victims have to make claims and so mitigation and sharing of civilians don't get harmed is the best place to be and maybe AI and algorithms will make small tiny precise weapons that we don't have to drop 2000 point bombs to kill one person but I also think that with the law itself we also need to rethink the law and suggest going to have written about the warification of IGL that there's this slide back towards justifying all violence as being proportionate as being necessary and more and really we've got lost sight of the fundamentals that war should be against our enemies and not against the people who aren't fighting us. So I think ideally you know remedy, redress, the man's it's about recognizing that other people are human and that we don't want them to get caught up on war. It's a way for those who are responsible for causing the violence not legally being responsible but for the factual causation of violence to take ownership and to say yes sorry we dropped that bomb that happens we use this system that gives the wrong output we will then recode the system in some ways with those nab and again. So mitigation is a way for a change of culture, a change of practice that's constantly via the rate of that does improve. My concern is that we're dealing with two very different systems that's one is database and quantitative and law is qualitative. It's about values. It's about human judgments. It's about reason. It's as you can't you can't teach a calculator to dance. I know it's just it's it's two two worlds do not collide. They may be representations of the others but it's it's abstract and removed from reality. And we start thinking we're going to do this some more and they could see if I made people not harmed it's going to be difficult. I think it's not impossible. If you're fighting a war right in the middle of Pacific and nobody's are you know then these sort of two is incredibly useful or in the desert or whatever but the practice of the last 20 30 years has shown that the algorithm of turn of war and using data has caused massive civilian harm where the machine said it was okay. So I just I'm very cautious. Cautious but optimistic. I mean that's a question to all of you. I think at the moment there does seem to be this real panic within our sector that we're not going to be able to keep up with the rate of development. I'll keep up with the rate but we're not we're almost always chasing things that have been developed you know long before we're maybe aware of them. But I mean yeah that's that's a question how optimistic are you that we can see a future where these types of systems are able to effectively reduce the number of civilians that are killed in conflict. I'm the eternal optimist and recently at the last session of the GGE laws the chair Robert and then both ambassador in the bus he mentioned he was a realist and he said a realist is an optimist with experience and I think that I really have kind of come to embrace that because I think there is a lot of maybe that perception of panic in our sector etc but I think also we should sometimes take a breath ourselves and realize look we have agency in this these are systems we are designing and developing and programming we're still in charge of them they're not in charge of us and it's not an inevitability that they will be so that makes it exciting for me and this is where my optimism comes in because I think because we're here at the ground floor because we are surrounding ourselves here at this particular meeting with all kinds of expertise and learning from one another if you can find those kinds of communities in places where people are open to listening and if you're open to asking questions and listening yourself to learning you know but I think that this is where we can build systems that ensure that continuous legal oversight the transparency cooperation between governments industry academia to make sure that that accountability keeps pace with the capability right and I know that the law is always the slow thing to catch up with technological developments but in this particular scenario I do think that the legitimacy of military AI depends on that shared responsibility but that also offers opportunities for anybody who's interested come and join the party and be part of the solution rather than the problem. Yeah absolutely I'll join and you know yeah agree about you know kind of that that sense of perhaps optimism but I think grounded in agency right and not a passive optimism not that things are either going to go not well or well as a bystander watches that happen in front of us but instead we have choices and we have agency in which direction that goes and in that sense you know I feel optimistic and I think another reason you know I feel that sense perhaps on this issue is because unlike some other issues that relate to you know protection of civilians or I.H.L. or some of the issues that we care about so much this also has reverberations across many other sectors and aspects of society and people are developing intuitions experiences with this technology thinking how it thinking through how it affects their lives how it affects their work how it affects their communities and I think you know that that is a ripe area from which to have more informed and thoughtful conversations about what it means for war what it means for the use of military force how do we want a technology like this to be used in our name and to that extent I think yeah I do feel optimistic. optimism because I think from that experience people will feel, I hope, a sense of agency in the sense that they can decide how they want to shape that. I agree. I think, you know, law is what we make of it. It's a very useful tool. I can do great harm, good bad. But we've got to respect that we're inter-Ishmanitarian law of arm-conflict came from. It was a crucible of violence in the past and then I need to restrain that. Because we've perhaps become complacent or it's been warped and which war should be fought in a way that's a measure of restraint. But there's a lot of effort now to think about even fighting war. How do you think of a pace? You don't have that mindset that war isn't the only option. It isn't just an option for solving all the problems. Law has a place in the value and limit those successes and we need to sort of not forget the hard one struggles that got us to this point that we have laws that attempt and we'll see its comply and have signed up to the Geneva Convention, whatever. But we also need to, you know, maybe stand up, get out there and start advocating and not thought we're not already in this, but you know, to be sort of thinking about, you know, where do we want to leave law for generations to come? And that we didn't just throw our arms and say, well, we can't deal with these new technologies. These are nerve-racking issues like Henry Deont, he started, you know, the Tarashkir, the Kralis, was right in an 1861 about new engines of war and about how if we don't deal with this as all murder, murder, murder and the same thing in the ICRC in 1938, it's got a convention on new engines of war. I think it's signed because of World War II happened, but this is always a challenge for each generation and we need to stand up and push back against legitimization of violence against others who aren't ourselves. So I think we have a fight ahead of us. I mean, I'm very encouraged by all of your optimism, but I really love what you said and I think, you know, fundamentally, I, Shell was about upholding humanity, right? And really, I think as we think about how we're using these systems when we're somehow taking the human nests out of it, it's really important that we keep that in the front of our minds, particularly as well, with engaging with communities that are on the receiving end of it. And I think a lot of this work, it is also so important to bring in those voices and to consult, you know, communities who are affected by the sharp end, Jess, as you said. Thank you again to our guest, Luke Moffert, Jessica Jalsey and Chris Rogers. And thank you for listening. We hope you've enjoyed this many series. To find out more about the civilian harm-addressed project by Queen ZCL and CIS Fire and for later updates, you can go to reparations.cub.ac.uk/civilian-arm.