pass@k

For code, whether a single answer is right is the wrong question — a developer can generate several attempts and run the tests. pass@k asks whether at least one of k samples passes, which measures something closer to how the model is actually used. It also rises with k for free, so the k has to be reported or the number means nothing.

Sample n ≥ k completions, count how many pass the tests, and estimate the probability that a random subset of size k contains one. Estimating from n > k rather than literally drawing k is not a detail: the naive estimator is badly biased at small n, and the correction is the reason the metric is trustworthy at all.

With c of n samples correct, pass@k = 1 − C(n−c, k)/C(n, k), the probability that a random k-subset misses every correct sample. It increases monotonically in k, so pass@1 and pass@100 are different metrics rather than the same one measured twice.