Why fairness metrics cannot all hold at once
Enforce one criterion (demographic parity, equal opportunity, equalized odds or predictive parity) and measure the others.
العربيةThe idea
The lab searches the group B threshold that satisfies the chosen criterion with A fixed, then reports the rest.
Data description
Synthetic data generated in your browser from stated distributions; no real personal data.
Suggested next experiment
Concepts in this experiment
Is one fairness metric enough?
No. Each metric answers a different question and some conflict mathematically when base rates differ. Compute several, justify the one that fits the context, and add a qualitative analysis of who is harmed by each error.
Impossibility results (نتائج الاستحالة في العدالة)
Kleinberg, Mullainathan & Raghavan (2016) and Chouldechova (2017) showed that with different base rates, calibration (or predictive parity) and equal error rates cannot all hold except in special cases. The criterion must be chosen and justified.
What is the COMPAS story?
COMPAS is a commercial recidivism risk tool. ProPublica (2016) found Black defendants who did not reoffend were labelled high risk more often (FPR gap); the vendor replied the tool was calibrated. Kleinberg and Chouldechova showed both can be true because the criteria cannot coexist with different base rates.
Scientific sources for this experiment
- Inherent Trade-Offs in the Fair Determination of Risk Scores (arXiv:1609.05807)J. Kleinberg, S. Mullainathan, M. Raghavan (2016) — Paper
- Fair prediction with disparate impact (arXiv:1610.07524)A. Chouldechova (2017) — Paper
- Machine BiasJ. Angwin, J. Larson, S. Mattu, L. Kirchner (ProPublica) (2016) — Documented case
