Code / Agents
verifiedAgentic Coding
Not a function from a description, but a task in a repository: read the issue, find the files, make the change, run the tests, fix what broke. The unit of work is a session rather than a completion, and what limits it is not whether the model can write the line — it usually can — but whether it can take fifty correct actions in a row.
The finding that reframed this work is that the interface matters as much as the model. The same model, given tools designed for it — a file viewer that pages, an edit command that validates syntax before committing, concise error output — succeeds far more often than one driving a raw shell. That is an argument about the environment, and it connects this to Long-Horizon Agent's problem rather than restating it: the horizon is long because the interface makes each step expensive, not only because the task is.
Success compounds multiplicatively, which is what makes long tasks hard out of proportion to their length. At 98% per action a fifty-action trajectory completes about a third of the time; at 95% it is under a tenth. So the lever with the most leverage is per-step reliability rather than model capability in the usual sense — and it is also why recovery matters more than accuracy, since an agent that notices and retries turns a failed step into a longer trajectory instead of a failed one.
failed-trajectories holds 17% of the budget; rest holds the remaining 83%.
Trajectories that fail at some step, against the ones that complete, in trajectories. Drag the task length up to watch failure take almost all of them — at 95% per action, fifty actions complete under a tenth of the time, which is why per-step reliability beats capability.
Reviewed by opendroid · 2026-08-18
- arXiv:2405.15793 — SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
- arXiv:2310.06770 — SWE-bench: Can Language Models Resolve Real-World GitHub Issues?