The eight fairness metrics calculator

Compute DP, SPD, DIR, equal opportunity, equalized odds, predictive parity and FPR/FNR differences from two confusion matrices, with the formulas filled in.

BeginnerAlgorithmic Fairness Lab
العربية

The idea

Enter TP, FP, TN and FN for each group. Each metric is shown with its definition, formula, numbers and interpretation.

Data description

Numbers you enter; defaults are a teaching example.

Suggested next experiment

Concepts in this experiment

Demographic parity formula

Gap between the highest and lowest selection rate; 0 means parity.

\mathrm{DPD}=\max_g P(\hat Y=1\mid G=g)-\min_g P(\hat Y=1\mid G=g)
Statistical parity difference formula

u = unprivileged, p = privileged; negative means u is selected less.

\mathrm{SPD}=P(\hat Y=1\mid G=u)-P(\hat Y=1\mid G=p)
Disparate impact ratio formula

Compared with the indicative four-fifths line 0.8.

\mathrm{DIR}=\frac{P(\hat Y=1\mid G=u)}{P(\hat Y=1\mid G=p)}
Equal opportunity difference formula

Difference in true positive rate between groups.

\mathrm{EOD}=\mathrm{TPR}_u-\mathrm{TPR}_p,\quad \mathrm{TPR}=\frac{TP}{TP+FN}
Equalized odds difference formula

The larger of the TPR and FPR gaps (Fairlearn definition).

\mathrm{EOdds}=\max\left(|\mathrm{TPR}_u-\mathrm{TPR}_p|,\;|\mathrm{FPR}_u-\mathrm{FPR}_p|\right)
Sensitive attribute

A characteristic that must not drive unjustified discrimination (gender, origin, religion, disability, age, depending on context and law). It is used in evaluation to measure group gaps even when not used for prediction.

Predictive parity difference formula

Difference in precision between groups.

\mathrm{PPD}=\mathrm{PPV}_u-\mathrm{PPV}_p,\quad \mathrm{PPV}=\frac{TP}{TP+FP}
Algorithmic fairness

Definitions and metrics of what it means for a system to treat groups fairly: independence (equal selection), separation (equal error rates given the true label) and sufficiency (equal predictive value). They cannot all hold when base rates differ, so choosing one is a contextual decision.

FPR and FNR differences formula

The u − p difference of each.

\mathrm{FPR}=\frac{FP}{FP+TN},\quad \mathrm{FNR}=\frac{FN}{FN+TP}
Demographic parity

Holds when the positive decision rate is equal across groups regardless of the true label. Measured by DPD (largest gap in selection rates) or DPR (ratio). Suits equal distribution of opportunities, but can conflict with accuracy when base rates differ.

Disparate impact ratio

Selection rate of the unprivileged group divided by that of the privileged group. The four-fifths (0.8) rule is an indicative guideline from US employment practice; below 0.8 warrants investigation but is not a legal verdict by itself.

Equal opportunity

Holds when the true positive rate (recall) is equal across groups: qualified people in each group get the opportunity at the same rate (Hardt, Price & Srebro, 2016). Suits contexts where denying the deserving is the main harm.

Equalized odds

Requires equal true positive and false positive rates across groups; the gap is usually the larger of the two differences. Stricter than equal opportunity because it covers both error types.

Predictive parity

Holds when the positive predictive value (precision) is equal across groups. Chouldechova showed it conflicts with equal error rates when base rates differ.

False positive / negative rate

FPR is the share of actual negatives wrongly flagged positive; FNR the share of actual positives missed. Group gaps in FPR or FNR mean the burden of error is unequal; the COMPAS debate centred on an FPR gap.

Scientific sources for this experiment

  1. Equality of Opportunity in Supervised Learning (arXiv:1610.02413)M. Hardt, E. Price, N. Srebro (2016) — Paper
  2. Fairlearn User Guide — AssessmentFairlearn contributors (2025) — Tool
  3. Fairness and Machine Learning: Limitations and OpportunitiesS. Barocas, M. Hardt, A. Narayanan (2023) — Book

More in this module