Structured Output

Sometimes the answer is not a sentence but an object: a parse tree, a table, an alignment, a JSON document a program will read. The difference that matters is that most possible outputs are not merely wrong, they are malformed — and a model that is 99% likely to produce well-formed output is unusable in a pipeline that runs a thousand times.

There are two families and they fail differently. Constrain the decoder so malformed output cannot be produced, and you get a guarantee at the cost of pushing the model off the distribution it was trained on. Ask nicely and retry, and you keep the distribution but have no guarantee, only a rate — which is fine at a hundred calls and not at a million. Which one is right depends entirely on what happens downstream when the output does not parse.

The reason this is hard is that the valid set is vanishingly small inside the space the decoder searches. With a vocabulary of tens of thousands and a sequence of hundreds, the number of token sequences is astronomically larger than the number that parse, and the ratio worsens with every additional token. So a model producing well-formed output at all is doing something remarkable — and asking it to do so reliably, rather than usually, is asking for a different kind of guarantee than a probability can give.