Conditional Computation

Most networks spend the same effort on every input, whether it is trivial or hard. Conditional computation lets the network decide, per input, which parts of itself to run. The appeal is obvious and the difficulty is specific: the decision is discrete, and you cannot take a gradient through a choice.

Any architecture where a learned gate switches parts of the graph on or off per example — early exit, skipped layers, routed experts. The blocker is that a hard gate is non-differentiable, so training needs an estimator: a straight-through pass that treats the discrete choice as identity on the backward pass, a continuous relaxation, or a policy-gradient treatment of the gate. A second failure mode is degenerate collapse, where the gate finds one branch and stops exploring.

With gate g(x) ∈ {0,1} and branch f, the forward pass computes y = g(x)·f(x) and ∂y/∂θ sub _g  is zero almost everywhere. The straight-through estimator substitutes the derivative of a surrogate, using ∂g̃/∂θ sub _g  with g̃ = σ(z) on the backward pass while the forward pass keeps the hard threshold. The bias this introduces is tolerated because the variance of an unbiased score-function estimator is worse in practice.