ML bias: Algorithms in the Courtroom!

What do we mean by bias in AI and what is at stake when AI is biased? Where is the source of AI bias? Is it in the data, in society, or in the analysis of the algorithm? And when it comes to fairness, should it prioritize equality in the outcomes or the procedures that create them? Welcome to Casecast, where we dive into another topic at the crossroad of AI and ethics. I'm your host, Yasar Arawani, a professor of Technology and Innovation at IV Business School. Today, we're diving into a topic that is both fascinating and, honestly, a little unsettling, machine learning bias. To help me untangle this complex and super important issue, I've got the perfect guest. My brilliant colleague, Dr. Lauren Ciprione, she is a professor in Management Science at IV Business School and the Canada Researcher in Healthcare Analytics Management and Policy. Lauren, welcome. Thank you, Yasar. It's great to be here. This topic hits close to home for me as I work in healthcare and health policy decision problems every day. Algorithms are changing the landscape of how patients interact with physicians and how physicians interact with the information about patients that have to make complex diagnosis and treatment decisions. Together, we'll explore the fascinating story of Compass, an algorithm used for risk assessment in the criminal justice system. Before hitting the road, let me define two terms that we're going to use frequently, algorithm versus moral. What do we mean by that? An algorithm refers to the specific computational method or procedure used to solve a problem or make a prediction, such as logistic regression or random forest. Algorithms are like recipes or methods that guide how to process data, identify patterns and make decisions. A model is a trained instance of an algorithm on a specific data set, producing a system of learned patterns. In the context of our today's podcast, Compass is a predictive model developed using machine learning techniques and algorithms like logistic regression is used to build Compass, moral to predict recidivism. Now, Lauren, before we dive in, how do you feel about the idea of algorithms weighing in on decisions that affect people's lives so directly? Honestly, it's both exciting and a bit unsettling. Algorithms can bring consistency and data-driven insights. Most of us are familiar with algorithms that recommend what to watch on streaming platforms, but the stakes are so low. Algorithms are quietly shaping our experiences in many ways that are high stakes, but that we don't see so obviously as our Netflix recommendations from selecting what news we see and the perspective it's presented in our newsfeed to determining our credit scores, deciding who gets a job interview. Models like Compass, which are used in the criminal justice system, have very serious implications, and so what we must question how fair and transparent they really are. Exactly. That's such a good point, Lauren. It's easy to think of morals and algorithms as neutral or objective, because they are based on data. But as we'll see with Compass, this story is far more complex. Compass stands for Correctional Offender Management Profiling for Alternative Sanctions. It is a tool developed by North Point Incorporation, now rebranded as equivalent, and has been used to assess over a million offenders since its inception in 1998. The way these models are designed, the data they're fed, the strengths and weaknesses of the algorithms used to build them and how they're used can have profound consequences, often in ways that reflect and amplify existing biases. And the idea is simple yet profound. Compass is an AI-based model used in the criminal justice system to assess the likelihood of a defendant reoffending, which is called recidivism. Essentially, it generates a risk score based on various factors, and this score is used by judges to make decisions about bail, sentencing, and even parole. It's meant to bring consistency to these decisions, but it raises significant questions about fairness and accountability. All right, let's dive in and uncover this story behind Compass. Predictive models in the criminal justice system, also known as Risk Assessment Tools, Rats, aimed to predict recidivism and aid decisions ranging from pretrial to peril. Compass is not the only one. Some widely used rats include the Public Safety Assessment, PSA, Virginia pretrial Risk Assessment Instrument, VPRAI, or Praxis in Michigan used to guide bail recommendations. But why bother with a model like Compass? Can't human judges with their experiences and judgments make better decisions without relying on a computer system? These tools aim to manage heavy workloads, cut costs, and reduce human bias. We worry about conflicts of interest or animus in the form of racism or discrimination against victims or defendants, but as humans, even factors like our mood or even the weather on a given day can influence our interactions and decisions. Unlike the human brain, algorithms can process numerous factors simultaneously and can be updated to reflect new laws and policy changes in ensuring consistency. The goal is to make judicial decisions more accurate, fair, and transparent than those based solely on human judgment, but it isn't clear that accuracy, fairness, and transparency are being achieved. Yes, actually, I've read about that. A study by Hayes and Sobreon found that a 10-degree Fahrenheit increase in outside temperature could reduce favorable rulings in immigration cases by 6.55%. It's fascinating and concerning how external factors can affect supposedly objective decisions like this. Historically, judges have relied on their past experience in intuition to estimate an individual's risk of reoffending when making crucial decisions like setting bail, granting probation or parole. When the stakes are so high, affecting both the lives of defendants and public safety, it's understandable why there's a push to use data-driven tools like Compass to support these decisions. As we're going to explore, these tools aren't without their own issues. And when seeing this opportunity in the late 1980s, Timothy Brennan, a professor at the University of Colorado, envisioned using AI for more objective risk assessments. He founded North Point Incorporation and developed Compass to predict recidivism risk using an algorithm trained on extensive criminal justice data. Lauren, what strikes you about this mission? It's ambitious, but also highlights the challenges. While automation with a data-driven model promises objectivity, the quality of its predictors depends heavily on the data and design choices made during development. His goal was to provide a more objective and accurate assessment of recidivism risk. If successful, this program could be more efficient and less prone to human bias in errors than traditional judicial intuition. Compass was designed to assist judges, probation officers, and other criminal justice professionals in making more informed decisions about defendants. It uses a vast array of data points to provide a risk score indicating the likelihood of a defendant committing another crime within two years. That sounds impressive, but it's important to note the complexity behind these kinds of algorithms. Compass uses data from a variety of sources, such as prior arrests, convictions, and even personal questionnaire answers. But we need to critically examine the development process to see whether these tools are delivering on that promise or if they just repackage bias in a more sophisticated way. Alright, let's shift gears and hit the fast track to model development. No matter the industry, understanding the development process is crucial. It is essential for effectively overseeing and managing AI systems. Compass is an AI system, as it is a mathematical model that mimics and enhances human predictions. In this case, it predicts re-engagement in criminal activity based on the data it receives. Absolutely, yes, sir. The development of AI systems like Compass involves several critical steps. It starts with identifying the problem in this case, predicting the likelihood of recidivism. Then comes the data collection phase, where vast amounts of information are gathered, often from historical records, surveys, or databases. This step is critical as the quality and representiveness of the data lay the foundation of the entire algorithm. This phase is more than just technical. It requires critical thinking to ensure the data represents the population the algorithm will serve. This is followed by data preprocessing where the raw data is cleaned, structured, and prepared for training the model. Once the data is ready, the actual mathematical model is developed, developers choose an algorithm type, train it using the preprocessed data, and fine tune it to improve its accuracy. It's worth noting that the choice of algorithm, whether a logistic regression, decision trees, or more advanced methods, can significantly affect the outcome and interpretability of the model. But here's the tricky part, the process doesn't end there. The algorithm must be rigorously tested to ensure it performs well across different scenarios and demographics. This is where questions about bias and fairness often come into the forefront, and finding solutions isn't always straightforward. And isn't one of the challenges ensuring that the data used to train the algorithm accurately reflects the population it will serve? Can truly unbiased data even exist? If the data is incomplete or biased, won't the algorithm inherit those flaws? Exactly. The source of the data and its quality are everything. If the training data over-represents or under-represents certain groups, the algorithm may produce biased results. For example, if police reports from over-policed communities dominate the data set, the algorithm could unfairly penalize individuals from those areas. It's the classic saying, "Gerbage in, garbage out." But with statistical and machine learning algorithms, spotting the garbage isn't always straightforward because bias can hide deep within these patterns. Without high quality, representative data, even the most advanced algorithms can yield biased or inaccurate results. And this raises another important question, how will developers even start collecting data in a way that minimizes these risks? To develop a functional model like Compass, Brennan started by gathering input and output data. This meant selecting a population, offenders from states like New York, Wisconsin, or Florida, for example. Then, he needed to identify the target output that he wanted to predict. In this case, the target output is whether the person re-offended, called recidivism, and they defined it precisely as whether the individual was re-arrested within two years. This is a great example of analyst choice. They could have chosen to define recidivism as within one year, three years or five years. They chose re-offending within two years. Even more, they could have chosen to define it broadly as any re-arrest, or narrowly as re-arrest for a violent crime, or very narrowly as re-incarceration. Each of these definitions would lead to very different prediction models and potentially different policy outcomes. So true, the final piece of assembling the data involves another set of choices. Brennan needed to identify variables available at the time the person is being sentenced to use as predictors of recidivism. That sounds complicated. How would someone trying to develop a model like this identify predicted variables? In a case like this, an interdisciplinary approach is critical. Bringing in experts from diverse fields ensures that the model captures nuanced and relevant variables. Ideally, a team consisting of sociologists, criminologists, and other social scientists, in addition to computer scientists, would collaborate to avoid overlooking critical factors and introducing human bias. Ultimately, the Compass model uses predictors from various sources within the criminal justice system, including demographic information, criminal history, social and personal background, and behavioral factors. I see two concerns here. First, demographic information, criminal history, social and personal background, and behavioral factors are very broad categories and lack specificity. Each one could involve multiple factors and every factor would need to be measured objectively. How did they ensure that these predictors were measured in a consistent and unbiased way? Second, what about the information they don't have? How do you ensure you're not missing critical variables that could improve the accuracy of predictions big time? Great question, and a big challenge for any AI project. After a hopefully multidisciplinary team determines the general concepts, specific measures for these general concepts need to be developed. For instance, Compass measured demographic information like age, gender, and ethnicity, but deliberately excluded race. Critics often point out that excluding race while including other variables like neighborhood or socioeconomic status can act that act as proxies for race reintroduces this bias indirectly. Let me give you some measures. For criminal history, they collected variables like the number of prior arrests and convictions, and the age at the first arrest. Personal background was measured through employment status and education level, while social relationships were assessed with questions like, was one of your parents ever sent to jail or prison, or how many of your friends are taking drugs illegally? Behavioral factors were measured with questions about past violence and attitudes towards criminal behavior, such as, how often did you get in fights while at a school, or to what extent you agree that a hungry person has a right to steal? This might be a very easy question to answer if you are not hungry. But obviously, defendants may seek to provide answers that would decrease the appearance of being considered an ongoing threat. These were some of the input variables. As you see, there are a lot of analyst choice in how to define each concept and what to provide as potential predictors. On one hand, there is the constraint of what is available. On the other hand, it seems like it really needs to be a thoughtful choice, doesn't it? Overall, for Compass, we don't know how many predictors or modified definitions of each predictor were even considered. Ultimately, the Northpoint model uses 137 predictor variables. From the fullest of predictor variables that were potentially available to them, the set of predictors actually used as identified through the modeling step. So let's recap. Northpoint might collect all the data on offenders in Florida for a specific period of time, identify whether they did or did not re-offend within two years, and then identify the full set of potential predictors for each person. Because many of the predictors Compass uses or answers to a questioner, they needed to find a setting in which the questioner they wanted to use had been widely adopted. You have it. Now, from the full set of potential predictors, we want to identify which subset of them are needed to create a high quality prediction model. When there are a lot of potential predictors available, it's easy to develop a prediction model that is overfit to the data. By that, I mean, it's great at predicting outcomes within the data it was built on, but it's not good at predicting in other data sets. A model that is only useful at doing prediction using the data on which it was built isn't very useful. The goal is that the prediction model is generalizable to other settings, you know, where the outcome isn't already known. So how do you make sure that your model is good at predictions in new data, not just in the data you have? If you want your model to be good at predicting using data, it didn't use for model selection and fitting. Then the model selection criterion needs to be evaluating exactly that. How good the model is at fitting out of sample data. So there are generally three steps to doing this very well. First, you split the data you have into a training set and a testing set. I usually use 70% of the data for training, but there's no correct answer about how much you should use for training and how much you should reserve for testing. After you have a preliminary model, you can compare how it performs on this testing data set. That's called internal validation. Second, when you are doing the mechanics of model selection and fitting, you can use a cross validation approach. When using cross validation, the training data itself is split into training and testing sets and candidate models are evaluated on whether or not they minimize prediction error in the testing set. Third, and this one's actually shockingly rare, but it's really a best practice. You collect a new fully independent data set and you test the model using that independent data set. This is called external validation. In this case, they could use a complete data set from Georgia or Pennsylvania. This is the truest test of when asking the question, is my model generalizable to other settings? You actually have to test it in those other settings. Whoa, that's that's an extensive process. So let me make sure we've got this right, Lauren. First, we split the data into two parts. Then we train the model on one part and test it on the other. Finally, we need at least one completely independent data set with the same predictors to evaluate the model externally. This gives us two levels of testing to ensure the model performs well on new data. Make sense, especially if this model is influencing decisions as serious as how long someone stays in prison. Did I get it right? Exactly. Yes, sir. Without rigorous testing, you risk deploying a model that only works on the data on which it was built, not in real world settings where it's meant to be applied. And when the stakes are high, generalizability is not negotiable. But now I'm curious, how does the actual model training work? You're the expert here, Lauren, or should I say the resident nerd break it down for me? I'll try and keep my nerdiness in check. The model training process depends on the type of algorithm chosen for prediction models, techniques like logistic regression, decision trees or ensemble methods like random forest are commonly used. The goal during training is to optimize an objective function like minimizing prediction error for recidivism risk. During optimization, the model explores and considers different relationships between inputs and outputs. Ultimately, these methods identify the patterns that link predictor variables to the target outcome. For instance, random forests create multiple decision trees using subsets of data, then average the results to improve accuracy and reduce overfitting. It's like having a committee of decision trees vote on the best prediction. Other methods like neural networks are more complex and can capture intricate relationships in the data, but they're also more prone to overfitting and harder to interpret. Each of these methods can be implemented different ways. An analyst can select predictors and model parameters in regression. These are the model coefficients using the full training data set or using the cross validation approach I mentioned earlier. When using cross validation, the training data itself is iteratively split into training and testing sets and candidate models are evaluated on whether they minimize the prediction error in the observations not used for training. All of this is automated so that hundreds and thousands of training and testing sets are created and the best model, including a specific subset of the available predictors, is selected based on how well it does at predicting outcomes in observations not used for training. One thing to note is that cross validation approaches are computationally very intensive. So different algorithms have different strengths and weaknesses, but what about compasses specifically? Compass is a proprietary model, so Northpoint hasn't disclosed the exact methods under the hood. However, based on available information, it likely relies on a combination of logistic regression and decision tree models. These methods are well suited for structured data like the variables, compass uses, and a law for relatively interpretable outputs like risk scores. But the lack of transparency is part of what makes compass controversial. It is hard to evaluate or challenge its predictions without knowing how it works. So the question is how can you trust something you can't fully understand? Exactly, and that brings us to one of the biggest challenges with algorithms like compass, balancing the promise of AI with the need for accountability and fairness. Transparency in how these models are built tested and deployed is crucial, especially when they're making decisions that deeply impact people's lives. Lauren, I still have the same question. Why should we trust a compass model? While compass doesn't reveal the details of the specific model structure, coefficients, or parameters, they do publicly provide the results of their validation analysis. During model evaluation, Northpoint focused on developing a model with high predictive value. For example, finding that the positive predicted value, which is the probability a person reoffends when they have been assigned a high score, or negative predictive value, which is the probability that somebody does not offend when they've been assigned a low score. Both of these are reasonable measures of accuracy, and that's what compass focused on. Here is what I found in my research about compass validation. In the early 2010s, independent researcher tested compass using historical criminal justice data, and found that it correctly predicted two-year recidivism between 64 and 74% of the time. Compared to 64% accuracy from human predictions. Northpoint's psychometric data underwent peer review in academic journals, and the company carried out multiple external validation studies nationwide. What was the result of external validation? Generally, compass performed on par with, if not slightly better than human decision makers, with less time required and fewer chances for individual bias. Northpoint promoted compass as an unbiased, data-driven tool designed solely to connect the dots within defendant information. This marketing implied to enterprise clients that compass would be objective and free from the emotional and learned biases of human operators. As a result, judges, probation officers, and other criminal justice professionals use compass scores to make crucial decisions about defendants. These scores contribute and influence the deliberation about who can be safely diverted to alternative programs, who should be released before trial, and who requires close monitoring or incarceration. But wait a second. What exactly did compasses spit out for judges to use? Compass presents a risk score on a scale from one to ten, corresponding with the probability of recidivism. In validation data sets, both internal and external, it has been shown that these risk scores are correlated with the likelihood of reoffending. However, note that not all tools present their results the same way. For example, that public safety assessment PSA, you spoke about earlier, provides multiple risk scores. One for the risk of failure to appear in court, another for the risk of new criminal activity, both on a scale of one to six. It even includes a binary variable, a simple yes or no, for the risk of new violent criminal activity. The Ohio risk assessment system categorizes risk levels as low, moderate, or high, so the format of the results can vary significantly depending on the tool. Alright, now that we've explored the details of how compass was developed, let's shift gears and discuss its deployment and real world application in the criminal justice system. Compass was launched in 1998 and was quickly adopted by a criminal justice system in New York, Wisconsin, and California, among other places across the United States. Marketed to course as an advanced tool to predict the likelihood of future crimes, Compass aimed to provide valuable insights for judicial decisions. Compass was enthusiastically embraced by many judges and used widely across various jurisdictions. Judges were instructed to use compass scores to inform their sentences and rulings, but not to solely base judgments on them. Interestingly, compass use was not mandatory, so some judges used it and some didn't use it at all or only used it in some cases. While the model was developed and trained on data of people convicted of crimes, some courts relied heavily on compass scores to inform decisions about bail prior to trial or parole decisions in addition to sentencing. This inconsistency undermined one of the stated goals of the tools, improving uniformity in judicial decision making. That's correct, Lauren. Judges varied in their use of compass. In some instances, it was used to determine the level of supervision a person received on peril. It was also used for analysis for pre-trial bail, peril, and probation decision making. Some agencies even relied on compass to help allocate resources and ensure the right supervision level. Here's an interesting example. In August 2013, a Wisconsin judge referenced Compass in the sentencing of the defendant, Eric Lumis, stating that the tool identified Lumis as a high-risk individual. The judge's statement about Lumis being at high risk to the community directly reflects the output of the compass assessment. This case led to a lawsuit and gained national attention and a spurred critique of recidivism risk assessment algorithms. It's fascinating how the adoption of compass varied so much, kind of reminds me of the diffusion of innovation theory by Everett Rogers. Oh, what's that, yes sir? It's a well-known theory that explains how innovations spread through a population. It's a framework with different components and arguments, but one key aspect is this. As an innovation spreads in user populations. It's used often evolves based on the needs, feedback, and context of the adopters. This adaptation can lead to modifications in the innovation itself, its application, or even how people perceive it. So the innovation doesn't stay static. It changes as it diffuses. Oh, I see. So as Compass spread, the way judges and agencies used it influenced and evolved, right? Exactly. And that leads us to another issue worth noting. A broader challenge with introducing technologies into the courtroom, the group to individual problem. I heard about this on David Gilman's podcast on courtroom technologies, where he explained the dilemma of using MRI scans in the courtroom. Scientific results from testing a technology often come from examining groups of people, but the justice system requires decision about individuals. For example, Compass uses patterns derived from historical group data to assess an individual's recidivism risk, but group averages can mislead when applied to individuals. Imagine research accurately identifies that men on average are taller than women, but then assumes a tall woman must be a man because her height matches the height of an average male. This issue raises critical questions about how specific and sensitive such tools need to be to truly inform individual decisions. That's such a great point, yes, sir. The group to individual problem is especially relevant here. Compass scores are calculated based on group level data, yet they're used to make high steaks decisions about individuals, bail, sentencing, and parole, where steaks couldn't be higher. Exactly. So that's not the end of Compass's story. In 2014, then US Attorney General Eric Holder warned that the risk score might be injecting bias into the courts. Later in May 2016, pro-publica, an investigative journalism organization published a bombshell critical report scrutinizing Compass. They found out despite not explicitly including race, other data aspects correlated with race, leading to racial disparities in predictions. Specifically, pro-publica reported that black defendants who did not re-offend were nearly twice as likely to be incorrectly predicted to re-offend comparing to white defendants, 45% to 23%. Conversely, white defendants who did not re-offend were almost twice as likely to be incorrectly predicted not to re-offend compared to black defendants, 48% versus 28%. That's why he's roaring. Break it down for us. What's going on here? However, pro-publica flipped the evaluation and took a more impact-oriented approach to the evaluation. They asked, among people who never re-offend, what was the probability of falsely being assigned a high-risk score? This is called a false positive. Among people who did re-offend, what is the probability of falsely being assigned a low-risk score? And among those assigned low scores, what percentage did re-offend? This is called a false negative. They examined whether these probabilities, the probability of being falsely assigned a positive score, or falsely assigned a negative score, were consistent across racial groups. Their analysis revealed that black defendants were disproportionately classified as high-risk when they did not re-offend. White defendants were more likely to be inaccurately assigned a low score. While technical metrics like rank order accuracy, matter-to-developers and researchers, false positives and negatives directly connect to how people experience the justice system. Other studies have also picked up some critical issues. One study specifically highlighted that compass reliance on arrest records as proxies for criminal behavior introduced significant biases, as arrest rates can be influenced by policing practices and racial profiling. This reliance on arrest record, rather than direct measures of re-offending pointed to potential bias. I know, and concerns arose about how compass was deployed in the judicial system. Judges were often influenced by the risk score in their sentencing decisions, sometimes leading to harsher penalties depending on risk scores, even if the individual circumstances didn't fully justify the outcomes. Transparency is key when implementing a system like this. Did North Point share their training data or their algorithm with others? Nope. As a propriory system, North Point did not disclose the inner workings of compass. However, researchers like Dressel and Ferrid found that compass prediction algorithm could be as simple as a linear classifier using just two features, age and the number of prior convictions, to achieve similar accuracy. Lauren, does it mean that those 137 variables is just fluff? It's very interesting that Dressel and Ferrid were able to find a model with similar performance characteristics to the compass model with just two predictors. In general, in statistics, a simpler model, we say parsimonious model is preferred. It's easier to understand, easier to validate, easier to explain, easier to deploy. But I have to say yes or a model with two predictors would make a terrible AI company. It would be easily reproducible, and no one would need your software to do the calculations for them. One of the barriers to adoption of AI tools is that their black box, which decreases public trust, expert trust, and policymakers trust in them. However, if adopted, a black box keeps the secret of the model indefinitely. I would also like to add that predicting something as complex as human recidivism, shaped by individual, familial, societal, and geographical factors, using just two variables is a massive oversimplification. Complexity theory teaches us that systems designed to govern or predict complex phenomena need to match their complexity. Over-simplified tools risk missing critical interdependencies and non-linear relationships. Whether we are making decisions or building models, our tools must be as nuanced, interconnected, and complex as the system they aim to address. Anyways, let me tell you Lauren what the company did. In response to a pro-publica report north-point defended their model, arguing that the observed disparities were a natural consequence of using on-bars as scoring rules on groups with different score distributions. They detailed extensive validation studies conducted both internally and by independent researchers, which consistently showed positive results and good predictive ability across various demographic groups, north-point asserted that their tool was equally predictive for both black and white defendants, with similar accuracy rates in both their studies and pro-publica's analysis. 62.5% for white defendants versus 62.3% for black defendants in pro-publica report, and 69% for white defendants versus 67% for black defendants in north-point study. Simplify this for us Lauren. Let's take a look at exhibit one in our case, where I've summarized the specific versions of the arguments in statistical language. North-point went back to the metrics used for model selection and validation. Conditional on being a reoffender, the model is equally good at giving higher scores to black and white people. Conditional on not being a reoffender, the model is equally good at giving lower scores to black and white people. But what this means in Broward County, where the overall recidivism rate for black defendants was 51% in comparison to 39% for white defendants, so the probability of being assigned a higher score was greater for black people than for white people. North-point emphasized that their model mirrored real world data rather than introducing new biases. But what pro-publica pointed out was still correct, conditional on being assigned a higher score, the probability that the higher score was incorrect is higher for black people than it is for white people. North-point doesn't disagree with that, but it says that this is only because of the difference in base rate of reoffending by race. Lauren, help me understand what the heck is base rate. Base rate is the underlying probability of the truth. In application, you don't know what the truth is, but in the training, testing, and validation data, you do know what the true rate of recidivism is. Because the base rate of recidivism is higher in black defendants, the model is going to assign higher scores on average to black defendants when it's deployed in the real world. This leads to another criticism of the model and of AM models in general with disparate outcomes by race. The racial disparities highlighted by pro-publica underscore broader systemic issues. For example, the higher arrest rates for black individuals, often driven by over-pleasing in certain communities, create feedback loops in data sets that perpetuate bias in predictive models. These disparities make it harder for individuals from marginalized groups to break free of the cycle of criminalization, further reinforcing social inequalities. In technical terms, Northpoint prioritized calibration, matching scores to recidivism probabilities over equal error rates across groups. So pro-publica's findings suggest that even a calibrated tool can lead to an equitable outcomes when applied in systems with historical and structural biases. This shift in focus from technical calibration to fairness in outcomes frames the broader ethical question. Should AI systems and the models they rely on mirror the inequities of the real world or should they strive to counteract them? That's the million dollar question. I should also mention that companies response also stress the importance of using compass as intended and aid in decision making rather than the sole determinant of judicial outcomes. US guidelines from the National Center for State course recommend using risk assessments to inform public safety considerations without relying on them exclusively for sentencing decisions. Northpoint also addresses the black box criticism, explaining that compass is a well-understood statistical model with theoretically justified features. They have made efforts to ensure transparency in the variables and scoring processes for agencies that use the tool, but they didn't really open the box. Pro-publica countered the criticism with another response and academic researchers also joined the debate. This controversy drew attention from various newspapers and the Wisconsin Supreme Court referenced it in their decision to uphold the use of compass in sentencing. This is an ongoing debate and it is relevant beyond sentencing guidelines. Similar algorithms are suggesting what news we read, what we watch for enjoyment and who we date. But also, whether we get selected for a job interview, whether we're issued a credit or approved for a mortgage, what mortgage rate we're going to pay, and even whether we qualify for certain medical treatments or insurance rates. These algorithms are woven into the fabric of modern decision making, affecting nearly every aspect of our lives. The compass controversy is particularly significant because it highlights the broader question, how do we ensure fairness, accountability, and transparency when algorithms influence decisions that carry such a high stakes? The debate isn't just about technical performance, whether the algorithm is accurate or calibrated. But about the values and ethics embedded within these tools, when systems like compass reflect existing societal biases, they risk perpetuating those biases in ways that are harder to detect and even harder to challenge. That's such a critical point, Laura. It raises the question of responsibility. Should we hold developers accountable for the outcomes their algorithms produce or is the responsibility shared with the institutions using these tools? After all, no algorithm exists in isolation. They're part of larger systems of decision making. Exactly. It's a shared responsibility. Developers must design and validate these tools with fairness and inclusivity in mind, while institutions must critically evaluate how they implement them. Blindly trusting a tool like compass or rejecting it outright misses the point. The real challenge is finding ways to leverage these technologies responsibly, ensuring that they complement human judgment without reinforcing inequalities. Lauren, it was wonderful to host you as the resident nerd on casecasts to discuss such an important topic. Thank you for having me, Yasser. All right. This story of compass highlights the kinds of challenges managers face in developing, deploying, and managing AI, such as loan approval system, mental health applications, recommendation systems, surge engines, fraud detection systems, predictive policing technology, HR analytics, healthcare robots, facial recognition technology, or autonomous vehicles. Let's take a moment to reflect on a few questions about AI bias before our class discussion. Do you side with ProPublica or Northpoint in this debate? What types of biases can you identify in the development and deployment of compass? Where are the locus of these biases? In the algorithm, in the data, or in the society at large, and as a manager, what steps would you take to mitigate any identified biases? These questions will guide us as we dive deeper into ethical and practical challenges organizations face, then developing and deploying AI system, also in class. [Music]