Privacy Lab
Personal data, anonymisation, pseudonymisation, differential privacy, privacy budget, membership inference and re-identification, on synthetic data only.
العربيةModule experiments (7)
Concepts and formulas
Laplace mechanism formula
Δf is the query sensitivity (1 for a count); smaller ε means more noise.
M(D)=f(D)+\mathrm{Lap}\!\left(\frac{\Delta f}{\varepsilon}\right),\quad \mathbb{E}|\text{noise}|=\frac{\Delta f}{\varepsilon}Definition of ε-differential privacy formula
For all datasets D, D′ differing in one person and all output sets S.
P[M(D)\in S]\le e^{\varepsilon}\,P[M(D^\prime)\in S]Sequential composition formula
The privacy budget is consumed by the sum of ε values.
\varepsilon_{\text{total}}=\sum_{i=1}^{k}\varepsilon_iPrivacy in AI
Protecting people from disclosure or misuse of information about them: data minimisation, purpose limitation, consent, anonymisation, and defence against model attacks such as membership inference and re-identification.
Membership advantage formula
0 means the attack is no better than guessing.
\mathrm{Adv}=\mathrm{TPR}_{\text{attack}}-\mathrm{FPR}_{\text{attack}}Personal data and quasi-identifiers
Personal data is any information about an identified or identifiable person. Direct identifiers single someone out alone; quasi-identifiers do so in combination (age, postcode, sex). Sensitive data (health, biometrics) is a special category.
k-anonymity
Every record matches at least k − 1 others on quasi-identifiers, via generalisation or suppression. It protects against identity linkage but not attribute disclosure when the sensitive value is uniform in a group.
Pseudonymisation
Replacing identifiers with tokens that reveal the person only with separately kept extra information. Unkeyed hashes can be reversed by guessing when values are few; keyed HMAC is better. Pseudonymised data remains personal data under GDPR.
Differential privacy
A mathematical guarantee that an analysis's output barely changes when one person is added or removed. Achieved by calibrated noise, e.g. the Laplace mechanism with scale Δ/ε. Smaller ε = stronger privacy, more noise (Dwork & Roth, 2014).
Privacy budget
Under sequential composition, ε adds up across queries on the same data: k queries at ε each consume kε. A total budget is set and split; each extra query shrinks the others' share and raises their noise.
Membership inference
Trying to tell whether a record was in a model's training data. The simplest form (Yeom et al., 2018) thresholds the loss: an overfit model has lower loss on its training examples. Risk is measured by advantage = TPR − FPR; regularisation and DP reduce it. In the lab it runs on synthetic data only.
Re-identification
Linking "anonymous" records to real people by matching quasi-identifiers with a public source. Sweeney (2000) showed ZIP code, birth date and sex single out a large share of the US population.
Questions and guides
What is a good value of epsilon?
There is no universal value. Small values (below 1) give strong guarantees with much noise; large values weaken them. It depends on data sensitivity, number of queries (total budget) and needed accuracy, and should be disclosed.
Does removing names anonymise data?
No. Quasi-identifiers together can single people out and allow linkage with public sources (Sweeney, 2000). Try the re-identification and k-anonymity experiments.
Module sources
- Regulation (EU) 2016/679 — General Data Protection RegulationEuropean Union (2016) — Law
- The Algorithmic Foundations of Differential PrivacyC. Dwork, A. Roth (2014) — Paper
- Simple Demographics Often Identify People UniquelyL. Sweeney (2000) — Paper
- Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting (arXiv:1709.01604)S. Yeom, I. Giacomelli, M. Fredrikson, S. Jha (2018) — Paper
- Membership Inference Attacks against Machine Learning Models (arXiv:1610.05820)R. Shokri, M. Stronati, C. Song, V. Shmatikov (2017) — Paper
