The eight fairness metrics calculator
Compute DP, SPD, DIR, equal opportunity, equalized odds, predictive parity and FPR/FNR differences from two confusion matrices, with the formulas filled in.
العربيةThe idea
Enter TP, FP, TN and FN for each group. Each metric is shown with its definition, formula, numbers and interpretation.
Data description
Numbers you enter; defaults are a teaching example.
Suggested next experiment
Concepts in this experiment
Demographic parity formula
Gap between the highest and lowest selection rate; 0 means parity.
\mathrm{DPD}=\max_g P(\hat Y=1\mid G=g)-\min_g P(\hat Y=1\mid G=g)Statistical parity difference formula
u = unprivileged, p = privileged; negative means u is selected less.
\mathrm{SPD}=P(\hat Y=1\mid G=u)-P(\hat Y=1\mid G=p)Disparate impact ratio formula
Compared with the indicative four-fifths line 0.8.
\mathrm{DIR}=\frac{P(\hat Y=1\mid G=u)}{P(\hat Y=1\mid G=p)}Equal opportunity difference formula
Difference in true positive rate between groups.
\mathrm{EOD}=\mathrm{TPR}_u-\mathrm{TPR}_p,\quad \mathrm{TPR}=\frac{TP}{TP+FN}Equalized odds difference formula
The larger of the TPR and FPR gaps (Fairlearn definition).
\mathrm{EOdds}=\max\left(|\mathrm{TPR}_u-\mathrm{TPR}_p|,\;|\mathrm{FPR}_u-\mathrm{FPR}_p|\right)Sensitive attribute
A characteristic that must not drive unjustified discrimination (gender, origin, religion, disability, age, depending on context and law). It is used in evaluation to measure group gaps even when not used for prediction.
Predictive parity difference formula
Difference in precision between groups.
\mathrm{PPD}=\mathrm{PPV}_u-\mathrm{PPV}_p,\quad \mathrm{PPV}=\frac{TP}{TP+FP}Algorithmic fairness
Definitions and metrics of what it means for a system to treat groups fairly: independence (equal selection), separation (equal error rates given the true label) and sufficiency (equal predictive value). They cannot all hold when base rates differ, so choosing one is a contextual decision.
FPR and FNR differences formula
The u − p difference of each.
\mathrm{FPR}=\frac{FP}{FP+TN},\quad \mathrm{FNR}=\frac{FN}{FN+TP}Demographic parity
Holds when the positive decision rate is equal across groups regardless of the true label. Measured by DPD (largest gap in selection rates) or DPR (ratio). Suits equal distribution of opportunities, but can conflict with accuracy when base rates differ.
Disparate impact ratio
Selection rate of the unprivileged group divided by that of the privileged group. The four-fifths (0.8) rule is an indicative guideline from US employment practice; below 0.8 warrants investigation but is not a legal verdict by itself.
Equal opportunity
Holds when the true positive rate (recall) is equal across groups: qualified people in each group get the opportunity at the same rate (Hardt, Price & Srebro, 2016). Suits contexts where denying the deserving is the main harm.
Equalized odds
Requires equal true positive and false positive rates across groups; the gap is usually the larger of the two differences. Stricter than equal opportunity because it covers both error types.
Predictive parity
Holds when the positive predictive value (precision) is equal across groups. Chouldechova showed it conflicts with equal error rates when base rates differ.
False positive / negative rate
FPR is the share of actual negatives wrongly flagged positive; FNR the share of actual positives missed. Group gaps in FPR or FNR mean the burden of error is unequal; the COMPAS debate centred on an FPR gap.
Scientific sources for this experiment
- Equality of Opportunity in Supervised Learning (arXiv:1610.02413)M. Hardt, E. Price, N. Srebro (2016) — Paper
- Fairlearn User Guide — AssessmentFairlearn contributors (2025) — Tool
- Fairness and Machine Learning: Limitations and OpportunitiesS. Barocas, M. Hardt, A. Narayanan (2023) — Book
