Code / Verification
verifiedTest Generation
If execution is how code gets verified, then tests are the bottleneck — and writing them is exactly the tedious work people want a model for. The circularity is obvious and worth stating: a model that misunderstands the specification writes tests that encode the misunderstanding, and then the wrong code passes them confidently.
The trick that makes it useful anyway is agreement rather than trust. Generate many programs and many tests independently, run every program against every test, and prefer the programs that agree with the most tests and the tests that agree with the most programs. Neither side is trusted individually; the consensus is the signal, and it works because independent errors rarely agree while correct implementations always do.
What a generated suite buys is coverage, and coverage saturates against the behaviours that matter. Each new test adds less than the last because the easy paths are exercised first and the interesting inputs — empty, enormous, adversarial, concurrent — are exactly the ones a model generating typical cases does not think of. So the curve rises steeply and then flattens well below the specification, and the remaining gap is not closed by generating more of the same.
generated-tests holds 12% of the budget; rest holds the remaining 88%.
Behaviours a generated suite exercises, against the ones it never reaches, in behaviours. Drag the test count up to watch coverage saturate below the specification — the inputs left over are the empty, enormous and adversarial ones a model writing typical cases does not think of.
Reviewed by opendroid · 2026-08-18
- arXiv:2207.10397 — CodeT: Code Generation with Generated Tests
- arXiv:2108.07732 — Program Synthesis with Large Language Models