Fill in the Middle

Real editing almost never happens at the end of a file. The cursor sits in the middle, with code above it and code below, and a left-to-right model can only see one of those. Fill-in-the-middle is the training change that lets a model condition on both — and it is what makes autocomplete inside an existing file work at all.

The transformation is disarmingly simple: take a fraction of training documents, cut them into prefix, middle and suffix, and reorder them so the middle comes last with the prefix and suffix as context. The model learns the new arrangement from sentinel tokens. Nothing about the architecture changes — the same causal model, trained on documents that have been rearranged, acquires an ability the architecture appeared to forbid.

The result that made it standard is that the ability is close to free. Applying the transformation to a large share of documents leaves left-to-right performance essentially unchanged while adding infilling outright, so the trade a reader would expect — capacity spent here comes from there — does not show up at the rates people actually use. It has to bind eventually, because the model has finite capacity and the rearranged documents are a different distribution; the finding is that the binding point is above where anyone needs to be.