Program Repair

Given code that fails a test, produce code that passes it. It is the most commercially useful shape of the coding problem, because a failing test is a real signal that arrives for free from CI — and it is the shape with the sharpest known failure mode, which is producing a patch that satisfies the test and is nonetheless wrong.

The literature calls it overfitting to the test, and it is not exotic: deleting the assertion passes the test, and so does special-casing the input the test uses. Search-based repair found these constantly, and a language model asked to make a test pass will find them too, faster. What separates a usable system is what else it checks — the rest of the suite, whether the patch is minimal, whether it touches code the failure implicates.

The risk scales with how weak the failing test is relative to the specification. A patch is accepted if it passes; among the patches that pass, the share that are actually correct is set by how much of the intended behaviour the suite pins down, and that share falls as the suite gets thinner. So a repair system evaluated only on the test it was given reports a number that improves as the tests get worse, which is exactly the wrong direction for a metric to move.