Privacy Lab

Personal data, anonymisation, pseudonymisation, differential privacy, privacy budget, membership inference and re-identification, on synthetic data only.

العربية

Module experiments (7)

Concepts and formulas

Laplace mechanism formula

Δf is the query sensitivity (1 for a count); smaller ε means more noise.

M(D)=f(D)+\mathrm{Lap}\!\left(\frac{\Delta f}{\varepsilon}\right),\quad \mathbb{E}|\text{noise}|=\frac{\Delta f}{\varepsilon}

Try it in: Differential privacy and epsilon

Definition of ε-differential privacy formula

For all datasets D, D′ differing in one person and all output sets S.

P[M(D)\in S]\le e^{\varepsilon}\,P[M(D^\prime)\in S]

Try it in: Differential privacy and epsilon

Sequential composition formula

The privacy budget is consumed by the sum of ε values.

\varepsilon_{\text{total}}=\sum_{i=1}^{k}\varepsilon_i

Try it in: Privacy budget

Privacy in AI

Protecting people from disclosure or misuse of information about them: data minimisation, purpose limitation, consent, anonymisation, and defence against model attacks such as membership inference and re-identification.

Try it in: Personal or anonymous data?

Membership advantage formula

0 means the attack is no better than guessing.

\mathrm{Adv}=\mathrm{TPR}_{\text{attack}}-\mathrm{FPR}_{\text{attack}}

Try it in: Membership inference (safe, educational)

Personal data and quasi-identifiers

Personal data is any information about an identified or identifiable person. Direct identifiers single someone out alone; quasi-identifiers do so in combination (age, postcode, sex). Sensitive data (health, biometrics) is a special category.

Try it in: Personal or anonymous data?

k-anonymity

Every record matches at least k − 1 others on quasi-identifiers, via generalisation or suppression. It protects against identity linkage but not attribute disclosure when the sensitive value is uniform in a group.

Try it in: Anonymisation by generalisation (k-anonymity)

Pseudonymisation

Replacing identifiers with tokens that reveal the person only with separately kept extra information. Unkeyed hashes can be reversed by guessing when values are few; keyed HMAC is better. Pseudonymised data remains personal data under GDPR.

Try it in: Pseudonymisation

Differential privacy

A mathematical guarantee that an analysis's output barely changes when one person is added or removed. Achieved by calibrated noise, e.g. the Laplace mechanism with scale Δ/ε. Smaller ε = stronger privacy, more noise (Dwork & Roth, 2014).

Try it in: Differential privacy and epsilon

Privacy budget

Under sequential composition, ε adds up across queries on the same data: k queries at ε each consume kε. A total budget is set and split; each extra query shrinks the others' share and raises their noise.

Try it in: Privacy budget

Membership inference

Trying to tell whether a record was in a model's training data. The simplest form (Yeom et al., 2018) thresholds the loss: an overfit model has lower loss on its training examples. Risk is measured by advantage = TPR − FPR; regularisation and DP reduce it. In the lab it runs on synthetic data only.

Try it in: Membership inference (safe, educational)

Re-identification

Linking "anonymous" records to real people by matching quasi-identifiers with a public source. Sweeney (2000) showed ZIP code, birth date and sex single out a large share of the US population.

Try it in: Re-identification by linkage

Questions and guides

What is a good value of epsilon?

There is no universal value. Small values (below 1) give strong guarantees with much noise; large values weaken them. It depends on data sensitivity, number of queries (total budget) and needed accuracy, and should be disclosed.

Try it in: Differential privacy and epsilon

Does removing names anonymise data?

No. Quasi-identifiers together can single people out and allow linkage with public sources (Sweeney, 2000). Try the re-identification and k-anonymity experiments.

Try it in: Re-identification by linkage

Module sources

  1. Regulation (EU) 2016/679 — General Data Protection RegulationEuropean Union (2016) — Law
  2. The Algorithmic Foundations of Differential PrivacyC. Dwork, A. Roth (2014) — Paper
  3. Simple Demographics Often Identify People UniquelyL. Sweeney (2000) — Paper
  4. Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting (arXiv:1709.01604)S. Yeom, I. Giacomelli, M. Fredrikson, S. Jha (2018) — Paper
  5. Membership Inference Attacks against Machine Learning Models (arXiv:1610.05820)R. Shokri, M. Stronati, C. Song, V. Shmatikov (2017) — Paper