Instance Segmentation

Semantic segmentation says these pixels are all 'person'. Instance segmentation says these are person one and those are person two, which is the difference between a picture that can be described and one that can be counted. Every application that needs a number — cells in a slide, items on a belt, people in a crowd — needs this and not the other.

The dominant approach bolts a mask head onto a detector: find boxes, then predict a binary mask inside each one. It works because the per-instance problem is easy once the instance has been isolated — the hard part was separating them, and the detector already did that. The detail that mattered was fixing the coordinate rounding between the feature grid and the box, which had been quietly discarding the alignment the masks depend on.

The cost structure follows from that design and is worth seeing plainly: the backbone runs once per image, and the mask head runs once per detection. On a photograph with three objects the backbone is nearly all the compute; on a crowd scene with a hundred, the mask heads are. So the same model has a per-image cost on some inputs and a per-object cost on others, which is a genuinely awkward property to serve behind a latency target.