Agent Evaluation

Scoring an agent is harder than scoring an answer, because the same task has many valid trajectories and the thing you care about is whether the world ended up right. Grading the transcript rewards agents that look industrious; grading the end state rewards agents that finished.

State-based grading — did the test pass, is the row in the database, does the file exist — is the only kind that resists gaming, and it requires an environment you can inspect. The second requirement is isolation, since an agent that can reach the internet can reach the answer. Reporting pass@1 alongside pass@k matters here more than elsewhere: an agent allowed to retry is a different system from one that is not.

Report success against a state predicate rather than a transcript similarity, and state the attempt budget, since success rises with attempts exactly as it does for jailbreaks. Variance is high at realistic task counts: a suite of a hundred tasks gives a standard error near 5 points, so a three-point difference between agents is noise unless it is repeated.