Representation Bias

If a group is a small fraction of the training data, the model has had less opportunity to learn them and will usually be worse at them. No malice and no protected attribute is required — a rare group is simply harder, in exactly the way any rare thing is harder, and the aggregate accuracy number will not mention it because the group is too small to move an average.

It shows up most starkly where data collection followed convenience. Object recognition systems trained on web images perform substantially worse on household items photographed in low-income countries than in high-income ones — the objects are the same objects; the photographs are not the photographs the model was trained on. Fixing it is a data problem before it is a modelling problem, and reweighting the loss helps a little where collecting more data helps a lot.

The damage compounds because the group's share of the data and its share of the error move in opposite directions. A group at a small fraction of the corpus contributes proportionally little to the training signal and proportionally little to the reported metric, so it is simultaneously the least learned and the least visible. That is the whole mechanism, and it means aggregate accuracy is structurally incapable of surfacing the problem.