Chunking

A retriever returns whatever unit you indexed, so choosing that unit decides what the model gets to read. Chunks too small lose the context that made a passage mean anything; too large and the sentence that answers the question arrives buried in text that does not. Nothing here is learned — it is a decision made before any model runs.

Split on structure where the document has it — headings, paragraphs, code blocks — rather than on a fixed token count, which cuts sentences in half. Overlapping windows recover some of what a hard boundary destroys, at the cost of storing the overlap twice. The unit retrieved and the unit embedded need not be the same: indexing a summary and returning the whole section is a common and useful split.

For a chunk of c tokens containing an answer span of a tokens, the share of retrieved context that is signal is a/c, and a mean-pooled embedding averages over all c — so a fixed span's contribution to the vector falls as 1/c. Larger chunks retrieve more reliably and represent less precisely, which is the entire trade written as one ratio.