Constrained Decoding

Asking a model for JSON and hoping is a coin flip at scale. Constrained decoding removes the choice: at each step, tokens that could not continue a valid structure are masked before sampling, so a malformed result is not unlikely — it is unreachable.

Compile the schema or grammar into a state machine and mask, at each position, the logits of every token no transition allows. The mask can be precomputed per state, so the overhead is closer to a lookup than to a parse. It constrains form and not content: a schema-valid answer can still be completely wrong, and it will now be confidently well-formatted about it.

Masking sets disallowed logits to −∞ before the softmax, so their probability is exactly zero rather than merely small, and the distribution is renormalised over what is left. A constrained model is therefore not the unconstrained model filtered — it is a different distribution, and one that can be walked into a valid string the original would never have produced.