When Algorithms Decide: What Does Fairness Mean?

You’re in the back of a classroom. Your teacher has to pick someone to stay after school and help clean up. She picks you. You think: “That’s not fair. Why me and not them?”

Now imagine the school buys a computer program to make that decision. You answer 50 questions about your grades, your behavior, your home life. The computer calculates a number. It says you should stay. The same thing happens to another kid, who got different answers, and they don’t have to stay.

You probably still think: “That’s not fair.” But now the question is harder. What exactly makes it unfair? Is it that you were treated worse than the other kid? Is it that the computer was wrong about you? Is it that a machine shouldn’t be making this decision at all?

These are the questions philosophers, computer scientists, and lawyers argue about when they talk about algorithmic fairness. Let’s look at what they’ve found.

The Case That Started the Debate

In 2016, a news organization called ProPublica published an article about a tool called COMPAS. Many US states use COMPAS to help judges decide who should be released from jail before trial. The tool asks people questions and gives them a “risk score” — how likely they are to commit another crime if released.

ProPublica found something disturbing. The tool made mistakes with Black and white defendants at about the same rate overall. But the kinds of mistakes were very different. Black defendants were much more likely to be falsely labeled high-risk — marked as dangerous when they actually wouldn’t reoffend. White defendants were more likely to be falsely labeled low-risk — marked as safe when they actually would reoffend.

The company that made COMPAS pushed back. They said the tool was fair because of a different measure: Black and white defendants who got the same score were equally likely to reoffend. If you were labeled high-risk, it meant the same thing regardless of your race.

Both sides had a reasonable point. But here’s the weird twist: computer scientists soon proved that you can’t satisfy both measures at the same time (unless the tool is perfect, which no tool is). If the rates of actual crime are different between groups — which they are, for complicated reasons involving history, policing, and poverty — then you have to choose. You can make the scores mean the same thing for both groups, or you can make the error rates equal for both groups. You can’t do both.

This is called the impossibility result. And it kicked off a huge debate. If fairness measures conflict, which one is really fairness? Or is the whole idea of measuring fairness with math itself the problem?

Treating Like Cases Alike

One way to think about fairness is: treat like cases alike. If two people are similar in the ways that matter, they should get similar treatment. This is a very old idea — you can find versions of it in Aristotle, in legal systems around the world, and in arguments on every school playground.

But “treat like cases alike” doesn’t settle the COMPAS dispute, because the two sides disagree about what “like” means. Are two people “alike” when they have the same risk score? Or are they “alike” when they’re equally likely to reoffend in reality?

Some philosophers argue that fairness is about epistemic things — about what the score tells you. If a “7” means something different for Black defendants than for white defendants (like, Black defendants with a 7 actually reoffend 70% of the time, but white defendants with a 7 only reoffend 50% of the time), then the score doesn’t “mean the same thing” for both groups. That seems unfair: you’re being judged by a number that carries different information depending on who you are.

Other philosophers focus on caring equally. They say fairness is about whether the people running the algorithm care as much about one group as another. If the algorithm makes more mistakes about Black defendants — mistakes that lead to them staying in jail — that suggests the system doesn’t care about them equally. The cost of being locked up when you shouldn’t be is huge, and if that cost falls more heavily on one group, something has gone wrong.

Still others point out that what counts as fair depends on what the algorithm is used for. If COMPAS were used to give people free job training instead of to lock them up, getting falsely labeled high-risk would be a benefit, not a punishment. A tool that’s unfair in one context might be fine in another. This suggests that fairness isn’t just about the numbers — it’s about what happens to people because of those numbers.

Who Are the “Likes” We’re Comparing?

Here’s another wrinkle. When philosophers test their ideas about fairness, they sometimes use made-up groups — people in Room A versus Room B. The idea is that if a principle works for any two groups, it should work for real ones like racial groups.

But is that true? Some philosophers say no. Real social groups like racial groups have histories. They’ve been treated differently for centuries. Poverty, policing, education, housing — all of these are shaped by race in ways that make the comparison to Room A and Room B misleading.

Consider this: an algorithm predicts who will be a successful computer programmer. It doesn’t know the applicants’ gender. But it learns that being a math major in college is a good predictor. Now ask: is “math major” really gender-neutral? If society steers boys toward math and girls away from it, then the algorithm might be using gender without knowing it. The “neutral” trait carries the history of who gets encouraged to study what.

This gets complicated fast. If race and gender are socially constructed — meaning they’re partly made up of things like where you live, how much money your family has, how police treat you — then removing the race label from data doesn’t make the algorithm “race-blind.” It still picks up on all the things that race is made of. Some philosophers argue that statistical fairness measures miss this point entirely.

Other Ways Algorithms Can Be Unfair

Not all complaints about algorithmic fairness are about comparing groups. Sometimes the problem is that the algorithm is just wrong about you.

Imagine a bank uses an algorithm to decide who gets loans. The training data happens to show that everyone named “Jamila” defaulted. There’s no real reason for this — it’s just a fluke in the data. But now you, Jamila, can’t get a loan even though you’d pay it back. Is that unfair? Some philosophers say the unfairness isn’t just the inaccuracy — it’s that the mistake systematically excludes you from important opportunities.

Another worry: algorithms work by making generalizations about groups and applying them to individuals. When you apply for a job, the algorithm might know that people from your neighborhood tend to quit quickly. But you specifically don’t. Is it fair to be judged by what’s true of your group, even if it’s not true of you?

This is an old problem. Police have used racial profiling for decades. Insurance companies use age and gender to set rates. Algorithms make these kinds of group-to-individual leaps much faster and on a much larger scale. The question is whether that’s ever fair, and if so, when.

Some people argue that machines shouldn’t be making certain decisions at all. Maybe fairness requires a human decision-maker — someone who can listen to your story, consider exceptions, and treat you as a person rather than a data point. Others point out that humans are biased too, sometimes more than algorithms. A study found that when judges could override an algorithm’s recommendation for bail, they set cash bail for Black defendants more often than for white defendants who got the same recommendation. The algorithm was actually less biased than the humans.

Finally, there’s the problem of explanation. If an algorithm decides something important about you — whether you get a loan, stay in jail, or get into a school — should you be able to find out why? Some philosophers say yes, because fair process requires that you understand the basis for decisions that affect you. Others say explanation matters because it lets you catch errors. But modern algorithms are often too complex for anyone — even their creators — to fully explain. Is it fair to be judged by a system you can’t understand?

The Data Problem

Even if you had a perfect definition of algorithmic fairness, you’d still have a problem: the data.

Sometimes the data is measured wrong. A company wants to hire reliable people. They can’t measure reliability directly, so they use letters of recommendation from past employers. But what if past employers are biased? What if they see female employees who miss work due to family emergencies as “unreliable” but make excuses for male employees who do the same? Then the data is skewed — and the algorithm built on it will be skewed too. “Bias in, bias out.”

Sometimes the data is non-representative. Facial recognition tools work less well for dark-skinned women than for light-skinned men. Why? Because the training data had too many light-skinned men and not enough dark-skinned women. Companies literally didn’t include enough people like you for the algorithm to learn your face. Is that unfair? Many people think so — especially when these tools are used by police or schools.

The hardest problem: accurate data about an unjust world. Imagine the data is perfectly measured and perfectly representative. It shows that Black applicants have less wealth and lower incomes than white applicants on average. This is true — because of centuries of discrimination, redlining, and unequal opportunity. Now a bank uses this accurate data to decide who gets loans, and it denies more Black applicants. The algorithm is right about the world. But it’s also perpetuating the injustice that created those patterns in the first place.

Some philosophers call this compounding injustice. The algorithm takes a world that was already unfair and makes it even harder to escape. Even if no one intended to be racist, the result is that historical wrongs continue into the future.

The Proxy Problem

Here’s a puzzle that connects many of these issues. Most laws say you can’t make decisions based on race, gender, or other “protected” traits. So algorithms are designed to ignore those traits.

But algorithms are good at finding proxies — other traits that stand in for the protected ones. ZIP code predicts race because of housing segregation. Being a math major predicts gender because of how kids are raised. An algorithm might never “know” your race or gender, but it can reconstruct them from other data. If it’s allowed to use those other traits, the prohibition on using race or gender is meaningless.

So what makes something a proxy? Philosophers disagree.

One view: a trait is a proxy if it’s strongly correlated with the protected trait. If most people from a certain college are women, then college attendance is a proxy for gender.

Another view: a trait is a proxy if it gets all its predictive power from the protected trait. If ZIP code predicts loan repayment only because it correlates with race, then it’s a proxy. But if ZIP code also predicts loan repayment because of wealth or local job opportunities, then maybe it’s not.

A third view: a trait is a proxy if it exists because of past discrimination. ZIP code predicts race because of housing discrimination. So it’s a proxy in a way that “height” (which also correlates with gender) might not be.

A fourth view: intentions matter. If someone deliberately uses a neutral trait to discriminate, it’s a proxy. If they don’t, it’s not.

These disagreements matter because they affect what algorithms are allowed to do — and who gets hurt by them.

The Big Question

Nobody has solved algorithmic fairness. Philosophers, computer scientists, and lawyers still argue about what fairness requires, which measures matter, and whether the whole approach of using math to measure fairness is misguided.

But here’s something important to notice: algorithms are mirrors. When we build them, we teach them from our data. That data comes from a world that’s already unfair in many ways. So algorithms don’t just create unfairness — they often reveal it. A loan algorithm that denies more Black applicants even without knowing race is showing us how deeply inequality runs in our society.

The question isn’t just “How do we make algorithms fair?” It’s also “What do we owe each other in a world where decisions are increasingly made by machines?” And that question doesn’t have a math problem for an answer. It has a human one.

Key Terms

Term	What it does in this debate
Algorithmic fairness	The question of whether and when using machine learning to make decisions about people is fair
Predictive parity	The idea that a risk score should mean the same thing for different groups
Equalized odds	The idea that error rates (false positives and false negatives) should be the same for different groups
Impossibility result	The proof that you can’t satisfy both predictive parity and equalized odds at the same time (unless the algorithm is perfect)
Proxy	A trait that stands in for a protected trait (like ZIP code standing in for race)
Compounding injustice	When an algorithm takes a world that’s already unfair and makes things worse by using accurate data about past injustice
Social construction	The idea that categories like race and gender are partly created by social practices, not just biology

Key People

ProPublica — The news organization that published the 2016 investigation showing COMPAS treated Black and white defendants differently
Benjamin Eidelson — A philosopher who argues that using group-based generalizations about individuals can be unfair, and that we should care about how algorithms perpetuate inequality
Lily Hu — A philosopher who argues that race and gender are socially constructed, meaning algorithms can’t simply ignore them by removing labels from data
Deborah Hellman — A legal scholar who argues that differences in error rates between groups are suggestive of unfairness but not proof of it, and that compounding injustice matters morally

Things to Think About

A school uses an algorithm to predict which students are likely to drop out. It’s more accurate for some students than others. Is that unfair? Does it matter why it’s less accurate for certain groups?
An algorithm that predicts job performance is trained on data from a company that has historically hired mostly men. The algorithm “learns” that being a man is correlated with job success. Should the company be allowed to use this algorithm? What if it removes gender from the data first?
A self-driving car has to choose between hitting a pedestrian or swerving and injuring the passenger. Who should decide how the car makes that choice? Should it be the same for everyone, or should it depend on who the pedestrian and passenger are?
Some people say algorithms can never be fair because they’re built by people in an unfair world. Others say algorithms can be more fair than humans because they don’t have the same biases. Which side do you find more convincing, and why?

Where This Shows Up

College admissions — Some universities use algorithms to help decide which applicants to accept, raising questions about fairness to students from different backgrounds
Criminal justice — Many US states use risk assessment tools to help decide bail, sentencing, and parole
Hiring — Companies use software to screen job applications, sometimes without knowing whether the software is biased
Social media — The algorithms that decide what content you see can reinforce stereotypes or spread misinformation, which some argue is a form of unfairness
Healthcare — Algorithms help decide who gets medical treatment, and they can be less accurate for certain racial groups
Everyday life — Credit scores, insurance rates, and even school grades increasingly involve algorithmic systems that affect people’s lives in ways they can’t see or understand