Softmax

Softmax turns a list of arbitrary scores into a list of probabilities: everything comes out positive and the whole thing sums to one. It is how a model converts an opinion of unequal strengths into a distribution it can sample from or average over. The largest score gets the most weight, but not all of it.

Exponentiate each score and divide by the sum of the exponentials. Implementations subtract the row maximum first — mathematically a no-op, numerically the difference between working and overflowing to infinity. A temperature divides the scores before exponentiating: below one it sharpens toward the argmax, above one it flattens toward uniform.

softmax(z) sub _i  = exp(z sub _i ) / Σ sub _j  exp(z sub _j ). With temperature τ, softmax(z/τ). As τ → 0 the output approaches a one-hot vector at argmax z; as τ → ∞ it approaches the uniform distribution. The Jacobian is diag(p) − ppᵀ, which is why the gradient vanishes once one entry dominates.