Semantic Segmentation

Label every pixel with what it belongs to — road, building, sky — without distinguishing one car from the car parked behind it. It is classification run at full resolution, and the difficulty is that the network which is good at recognising things is good at it precisely because it threw the resolution away.

A classification backbone downsamples aggressively, typically by 32, because that is what lets deep layers see enough context to know what they are looking at. Then the output has to come back to full resolution. Fully convolutional networks got there by upsampling and adding skip connections from earlier, finer layers; dilated convolutions instead keep the resolution and widen the receptive field by spacing the kernel out. Both are answers to the same tension, and the tension does not go away.

At stride s the network makes one prediction per s² pixels, so a stride of 32 means one genuine decision covering 1,024 pixels and everything else interpolated between them. The interpolated count rises as the square of the stride while its share saturates toward the whole, which is why boundaries are where these models fail and why average pixel accuracy flatters them: boundary pixels are a small fraction of the image and almost all of the errors a person notices.