Tools / Interface
verifiedComputer Use
Instead of calling an API, drive the screen: look at a screenshot, move the mouse, type. It works on software with no API at all, which is most software, and it is far less reliable than a function call because the interface was designed for hands and eyes rather than for a model.
Grounding is the bottleneck — turning "the submit button" into coordinates — and it degrades with resolution, scrolling and anything that moves. Accessibility trees are more reliable than pixels where they exist, which makes them the first thing to try rather than a fallback. Success rates on realistic desktop benchmarks remain far below what the same models achieve on API tasks, and the gap is grounding rather than reasoning.
A task of n steps completes only if every step does, so end-to-end success is roughly the per-step rate to the nth power: 95% per step is 60% over ten steps and 36% over twenty. That exponent is why recovery matters more than accuracy — an agent that can detect and retry a failed step converts a product into a sum.
grounded-steps holds 50% of the budget; rest holds the remaining 50%.
Steps where the target was located correctly against steps where it was not, in steps. Drag the grounding up to watch reliability climb — and note the end-to-end rate is a product of these, not an average.
Reviewed by opendroid · 2026-08-18
- arXiv:2404.07972 — OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
- arXiv:2307.13854 — WebArena: A Realistic Web Environment for Building Autonomous Agents