Residual Connection

Instead of asking a layer to produce its output from scratch, a residual connection asks it only for the change, and adds that to what came in. A layer with nothing useful to contribute can output approximately zero and do no harm, which is what makes stacking a hundred of them survivable.

Output x + F(x) rather than F(x). The addition gives the gradient an unobstructed path to every earlier layer — it flows through the identity term whatever F does — which is what removes the degradation that made very deep plain networks train worse than shallow ones. In a transformer, every attention and feed-forward sub-layer sits inside one.

y = x + F(x), so ∂y/∂x = I + ∂F/∂x. The identity term keeps the Jacobian away from zero regardless of F's own gradient, so the product across L layers does not decay geometrically. This is the mechanism behind trainability at depth, not merely an optimisation trick.