Code Generation

Write the function from its description. Code is the domain where the field's usual evaluation problem goes away, because you can run the answer — there is no need for a human judge or a similarity score when the program either produces the right output or does not. That single property is why so much recent progress routes through code.

The evaluation that stuck is functional correctness against a test suite, not similarity to a reference solution. Similarity metrics were tried and are poorly correlated with whether the program works: two correct implementations can share almost no tokens, and a one-character change that breaks everything barely moves the score. Sampling many candidates and keeping any that passes is standard, which is why the metric has a k in it.

pass@k rises with k and saturates, and the shape is the useful part. With a per-sample success probability p, the chance that at least one of k samples works is 1 − (1 − p) super ^k  — steep while p·k is small and flat once it is not. So a model at 20% per sample reaches about two thirds by k = 5 and gains very little after 20, which means quoting pass@100 for a model that will be sampled once in production describes a system nobody is running.