In-Context Learning

A model can be shown a few examples in its prompt and then do the task, with no gradient step and no weight change. Nothing was trained; the examples simply condition what comes next. This is why one model serves a thousand tasks, and why prompt engineering became a job.

Few-shot prompting: put k demonstrations in the context and the model continues the pattern. It improves sharply with scale and is sensitive to things that ought not to matter — example order, formatting, and label wording all move accuracy. That sensitivity is the standing argument against treating a single prompt's score as the model's ability.

Conditioning rather than optimisation: p(y | x, D) with D the demonstrations, no update to θ. The cost is a longer context, so k demonstrations cost Θ(k) tokens and attention over them costs Θ((k·m)²) for examples of length m — which is why the technique meets its limit at the context window.