Dilated Convolution

A convolution sees a few neighbouring steps. Skip steps between the taps and it sees a wider span at the same cost, and stacking layers whose skips double each time gives a view that grows exponentially with depth. It is how convolutions reached long sequences before attention did, and it is still how most audio models see.

Receptive field is a design calculation rather than something learned: with kernel size k and dilations doubling, L layers reach roughly k·2 super ^L  steps. The failure mode is gridding — with a fixed dilation, some input positions are never read by any tap, so the doubling schedule is load-bearing rather than a convention. Causal padding keeps it usable for forecasting and generation.

y sub _t  = Σ sub _i  w sub _i  · x sub _{t − d·i}  for dilation d. Stacking layers with d = 1, 2, 4, … 2 super ^L−1  gives a receptive field of (k−1)(2 super ^L  − 1) + 1, exponential in depth at linear cost. Every position in that span is covered exactly when the dilation schedule doubles; skip a factor and the coverage develops holes.