Interpretability / Methods
verifiedProbing Classifier
Train a small classifier on a model's internal activations to predict some property — part of speech, sentiment, whether a position is winning. If it succeeds, the property is available in there. The catch, which took the field a while to take seriously, is that a powerful enough probe can learn the property itself rather than find it.
The remedy is a control task: the same probe, same capacity, trained on randomised labels. A probe that scores well on real labels and badly on random ones read something; one that scores well on both merely memorised. Selectivity — the gap between the two — is the number to report, and reporting accuracy alone is why early probing results overstated what models encode.
Probe accuracy bounds nothing on its own: with enough capacity a probe fits the training labels whatever the representation contains, so a high number says as much about the probe as about the model. Selectivity = accuracy(real) − accuracy(control) restores the comparison by holding capacity fixed across both runs, which is the only reason the difference carries information about the representation rather than about the probe.
Loss over 1000 training steps, starting near 7.2. It falls to about 1.95, with 93% of the total improvement arriving in the first half. A second line shows held-out, ending higher at about 2.35.
A probe's loss against the same probe measured on data it did not fit. Drag the probe capacity up to watch it drive the fitted curve down while the held-out line trails it — that gap is what separates a probe that read the representation from one that learned the task.
Reviewed by opendroid · 2026-08-18
- arXiv:1909.03368 — Designing and Interpreting Probes with Control Tasks
- arXiv:2102.12452 — Probing Classifiers: Promises, Shortcomings, and Advances