Privacy / Attacks
verifiedMembership Inference
Given a trained model and a record, decide whether that record was in the training set. It sounds modest and is not: membership in a medical study is itself the sensitive fact. It is also the attack that operationalises overfitting, because a model that is more confident on data it has seen is a model that answers this question for you.
The signal is usually loss or confidence, calibrated against shadow models trained on data from the same distribution. Average-case accuracy is a poor way to report it — an attack that is barely better than chance overall can be nearly certain on the most vulnerable records, and those are the ones that matter. Reporting true positive rate at low false positive rate is the fix the field converged on.
The attacker distinguishes p(loss | member) from p(loss | non-member), and the gap between them is bounded by the model's generalisation gap — which is why overfitting and privacy leakage are the same quantity seen from two sides. Differential privacy bounds this attack's advantage directly, at e super ε , which is what makes ε interpretable as something other than a knob.
Loss over 2000 training steps, starting near 7.2. It falls to about 1.95, with 93% of the total improvement arriving in the first half. A second line shows held-out, ending higher at about 2.15.
Loss on records the model trained on against loss on records it never saw. Drag the generalisation gap up to watch the two curves part — that gap is exactly what a membership attack reads, which is why overfitting and leakage are one quantity.
Reviewed by opendroid · 2026-08-18
- arXiv:1610.05820 — Membership Inference Attacks against Machine Learning Models