Can a Computer Be Fair? The Battle Over Algorithmic Fairness
When a Formula Decided Who Stays in Jail
In 2016, reporters at the website ProPublica dug into a tool called COMPAS — a computer program used by many U.S. courts to predict whether someone accused of a crime would commit another crime if released. A judge might see a defendant’s risk score and set bail high or low, or deny bail altogether. ProPublica found something troubling: the algorithm was making different kinds of mistakes for Black defendants and white defendants. Black people who never went on to commit another crime were nearly twice as likely as white people to be wrongly labeled high risk. At the same time, when the program gave a score of “high risk,” the actual chance that a person would reoffend was the same no matter their race.
So was COMPAS fair? It depends on which numbers you think matter. The software’s creators used a measure called predictive parity — a math rule that says if two people get the same risk score, the odds they’ll actually reoffend must be equal across racial groups. ProPublica focused on equalized odds, a different measure. Equalized odds demands that the rate of two types of mistakes — false alarms (someone safe called dangerous) and missed dangers (someone dangerous called safe) — be the same for each group. Both rules sound like they capture something important about fairness, yet computer scientists quickly proved a surprising result: in almost all real-world situations, an algorithm can’t satisfy both at the same time. That’s known as the Kleinberg–Chouldechova impossibility theorem (named after researchers who published it in 2016). The only way to have both is if every prediction is perfectly accurate, which real algorithms never are.
The reason lies in a basic fact: the base rate — the true percentage of people who go on to reoffend — often differs between groups. When the base rate for Black defendants and white defendants isn’t identical, the algorithm can’t be equal in both predictive parity and equalized odds. You have to choose which fairness yardstick you’ll use. But that choice is not just a math problem; it’s a moral one. What does fairness actually demand when machines make life-altering decisions?
What Does “Fairness” Even Mean?
If you ask many philosophers, fairness starts with a simple idea: treat like cases alike. Two people who are similar in all the ways that matter should be treated similarly. But that old rule doesn’t tell you which features make two cases alike. The COMPAS fight shows why. One side says the relevant “likeness” is the score’s meaning: a risk score of 7 should mean “7 out of 10 will reoffend” for every racial group. The other side says the likeness that counts is how often the algorithm gets it wrong for each person: a mistake that lands an innocent Black person in jail and a mistake that lands an innocent white person in jail should happen at the same rate.
Philosophers have come down on both sides. Brian Hedden (in a 2021 paper) argues that fairness fails if a score has different evidential support for different groups — if a Black person and a white person with the same score aren’t equally likely to actually reoffend. His measure, calibration within groups, requires that the score mean exactly the same thing for everyone. Benjamin Eva (2022) takes a similar epistemic approach — fairness is about accuracy. He says a score must reflect the real base rate in each group; otherwise, the algorithm hides real differences and treats people as equally risky when they aren’t.
Other thinkers shift the focus from knowing to caring. Boris Babic and Zoë Johnson King (2025) argue that fairness requires the people using the algorithm to care equally about each group. That care shows up in the balance of false alarms and missed dangers — the ratio of mistakes must not tilt against a disadvantaged group. Seth Lazar and Jake Stone (2023) add that we especially shouldn’t tolerate an algorithm that does worse for a group already facing background injustice when we could improve it without large losses for others.
Notice what all these views share: they see fairness as a comparative idea — you judge how one group is treated only by comparing it to another. But not every complaint about algorithmic unfairness is comparative. Sometimes a single person says, “I was treated worse than I should have been,” without reference to anyone else.
Fairness Beyond Numbers: When the Algorithm Feels Wrong
Imagine a bank uses a machine-learning tool to decide who gets a loan. The training data just happens to contain several people named Jamila who defaulted. The algorithm learns that the name “Jamila” predicts risk, even though there’s no sensible connection. When you — a new Jamila — apply, you get rejected. You aren’t being compared to anyone else; you just got treated worse than you ought to have been. Kathleen Creel and Deborah Hellman (2022) call this sort of arbitrariness not inherently unfair, but it can become unfair when it excludes people systemically from important opportunities, like every lender using the same data oddity.
Stereotyping raises a related worry. Many algorithms work by making predictions about you based on what’s true of people who share your group traits — your ZIP code, your education, your patterns of behavior. Is that any different from a human boss who assumes you’ll be unreliable because you’re a certain age or race? Some philosophers, like Benjamin Eidelson (2020), worry that using statistical generalizations robs you of being judged as an individual. Others, like Frederick Schauer (2003), point out that ignoring group information can sometimes be unfair, too — like treating all airplane passengers exactly the same when some pose a greater security risk.
Then there’s the complaint that only a human should make life-changing decisions. Even if an algorithm makes fewer mistakes than a judge, you might feel you deserve a person who can look you in the eye. Linda Eggert (2025) argues that machines lack moral receptiveness — the ability to appreciate the moral weight of their actions. Others highlight that fair process might demand you can challenge the result and explain your side, which a computer screen can’t hear.
Finally, when a complex algorithm decides your fate, you often can’t understand why. That inscrutability feels unfair. Maybe fairness requires an explanation — not just to catch errors but because being treated as a person means knowing why a decision was made. Kate Vredenburgh (2022) calls this “informed self-advocacy”: you need to know the factors so you can shape your future choices and correct mistakes.
The Dirty Data Problem
Algorithms don’t start out smart; they learn from data we feed them. If that data is skewed, the algorithm inherits the skew. Measurement error happens when we measure the wrong thing. Courts want to know who will commit a future crime, but they measure arrests. Arrests reflect both actual crime and police choices to patrol some neighborhoods more heavily. If certain racial groups are policed disproportionately, arrest records don’t capture true offending rates — the error hits some groups harder than others. As Sandra Mayson (2017) warns, even accurate-seeming inputs can smuggle in bias.
Data can also simply underrepresent people. In 2018, Joy Buolamwini and Timnit Gebru showed that top facial recognition systems recognized light-skinned men far more accurately than dark-skinned women. The training data was full of mostly white, mostly male faces. A representative sample might still contain too few minority group members for the tool to work equally well for them. When a service works worse for you because you’re from a smaller group, many argue that’s a distinct kind of unfairness.
Even when data is perfectly accurate, it can be unfair. Suppose a bank’s algorithm predicts loan repayment using income and savings. Black applicants, on average, have less wealth — the result of decades of discrimination. If the bank uses that fact, it may compound past injustice. Anikó Wächter and Deborah Hellman (2021, 2024) argue that when an actor relies on accurate data that reflects prior wrongdoing, they carry that unfairness forward — unless they have strong reasons to do otherwise. Algorithms can act like a mirror, reflecting society’s deep inequalities back at us with uncomfortable clarity.
Hiding Race in the Code: The Proxy Problem
Most legal systems forbid using race or sex to make hiring, lending, or sentencing decisions. Yet machine-learning tools can easily find other traits that work almost as well. A lender’s algorithm, unaware of race, might notice that ZIP codes predict repayment — and because of housing segregation, ZIP code is tightly linked to race. The algorithm ends up having the same effect without ever seeing a racial label. This is called the proxy problem.
What makes some trait a proxy for a protected characteristic? One view says: a strong correlation. If graduating from certain women’s colleges almost guarantees you’re female, then filtering by that college looks like filtering by sex. But a pure correlation test is tricky. In a famous U.S. Supreme Court case about veteran preference in hiring, the court said veteran status wasn’t a proxy for sex — even though almost all veterans were male — because the group of non-veterans included both men and women.
Another approach focuses on information value: a trait is a proxy if it predicts the outcome only because it correlates with the protected trait. If graduating from a women’s college predicts job performance solely because of sex, then it’s a proxy. But this test misses cases like ZIP code, which predicts loan repayment partly due to wealth and partly due to race — so it might not count.
A third answer traces the history behind the link. If housing discrimination and segregation caused ZIP codes to align with race, then ZIP code becomes a proxy — even if it also captures genuine economic differences. Annette Zimmermann and others explore this explanatory idea. A fourth view, prominent in law, says it’s the intention that matters: a trait becomes a proxy when someone deliberately uses it to select based on the protected characteristic. This intentional test makes it hard to call an algorithm’s pattern a proxy if the programmers never meant to discriminate.
The proxy debate is crucial because forbidding race directly will mean little if algorithms can always find a roundabout path to the same result. Deciding which work-arounds count as unfair forces us to settle what kind of justice we want — case law and moral philosophy are still wrestling with it.
Why This Fight Matters to You
You’ve probably been scored by an algorithm already — maybe a reading-level placement test, a social media feed, or a game that adjusts difficulty. In a few years, you’ll encounter tools that help decide which college you can attend, what jobs you see advertised, or even how much you pay for car insurance. Every one of those systems is built on some idea of fairness, even if the programmers never wrote the word down.
The COMPAS story shows why it’s a mistake to ask simply “is this algorithm fair?” as though fairness is a single switch that’s either on or off. You have to ask: which fairness? The one that says a score must mean the same thing for everyone? The one that says mistakes must fall equally? The one that demands you can look a human in the eye when your fate is decided? In the real world, these goals conflict, and choosing between them is a moral choice — not a technical one.
So next time you hear that a computer made a decision, don’t just wonder if it was accurate. Wonder what fairness standard it was aiming for, and who decided that. That question — as old as philosophy itself — is now woven into the code that runs your world.
Think about it
- If a court uses an algorithm that makes fewer overall mistakes than a human judge but makes more mistakes for one racial group, should it still be used? Why or why not?
- Imagine a school admission algorithm knows your hobbies, the books you borrowed from the library, and your neighborhood. Could it ever decide “more fairly” than a teacher who knows you personally? What might get lost?
- If a bank cannot legally use race, should it also be forbidden to use ZIP code? Where would you draw the line between a neutral fact and a hidden proxy — and would your rule change if you were building the algorithm for a friend?
