VerifierBench

Introduction

An agent that can solve a task cannot necessarily judge whether a solution to that task is correct. VerifierBench measures the second skill: given the same instruction and environment used for solving, the agent writes an executable verifier, and we test that verifier on hidden positive and negative cases.

Data-oriented recursive self-improvement needs agents to judge candidate solutions while building their own training data. An incorrect verifier can repeatedly reward a violation or discard a valid solution. A verifier has to tell genuine violations apart from allowed differences in tool use, representation, or implementation.

Benchmark

Tasks come from open-source agent benchmarks that have executable evaluation procedures. They cover three forms of verification: tool-call tasks check functions and arguments, state tasks check interactions or the resulting environment state, and code tasks check proposed software changes.

Each task starts from the reference solution (gold) and an unrepaired outcome (base). We then construct further cases and label each one PASS or FAIL from the task requirements before running the source verifier. Positive variations change presentation, implementation, or execution path while still satisfying the task. Negative cases include partial solutions, near misses, regressions, and 798 attacks that target checking omissions or fake success signals.

PASS and FAIL are reference labels. Alt. counts cosmetic variations and valid alternative solutions (a subset of PASS); Attack counts hack and process_violation cases (a subset of FAIL).

Reference Validation (V4V)

Source benchmarks ship reference verifiers and evaluation materials that can themselves be wrong. Before freezing the benchmark, Verifier4Verifier screens every reference verifier and test suite with six checks.

1. ReplayThe reference verifier's decision on each case matches the label assigned during construction.
2. RepeatabilityRepeated runs from the same reset state return the same decision.
3. ProvenanceEach recorded decision is tied to the exact case contents and reference version.
4. Input–label consistencyNo two cases within a task show the verifier identical information but carry opposite labels.
5. Trivial baselinesAlways-accept, always-reject, and exact-match verifiers are each exposed by the suite.
6. Mutation testingWeakening a check in the reference changes at least one valid decision; 1,410 task–operator configurations in total.

Confirmed defects in source materials led to 45 exclusions from the initial 267 tasks. Evaluation-pipeline errors were corrected without excluding tasks. The frozen records mark all six checks as passed for the 222 retained tasks.

Main Results

The primary metric is normalized task-averaged balanced accuracy, BA_norm = TPR + TNR − 100. Each task counts equally and each class counts equally within a task. A constant verifier scores 0 and perfect agreement with the reference scores 100. Missing or invalid outputs stay in the denominator with no credit.

Accepting valid solutions is the shared weakness. For all 19 pi-harness models, TPR (21.1–70.2%) is lower than TNR (79.2–96.3%). Explicit FAIL decisions account for 57.1–100% of each model's unaccepted positives; the rest got no valid decision.

Leaderboard

All configurations are scored on the same 222 tasks. Harnesses differ in prompts, tools, and reasoning settings, so this is a ranking of configurations, not an isolated comparison of models. FPR_h is the task-averaged acceptance rate on the 798 attack cases; lower is better. Click a column header to sort.

Native Harnesses

Switching from the pi harness to a model's native harness helps some models and hurts others. Claude Code raises Opus 4.8 by 12.2 points and Gemini CLI raises Gemini 3.5 Flash by 15.0, while Codex lowers all four GPT models by 4.1–26.6 points. In paired traces, some GPT-5.5/Codex verifiers demand particular patch contents or repair locations before running any functional check.

Where Errors Concentrate

False rejections are not spread thinly across tasks. For nine models with archived task-level records, verifiers that return FAIL on every case of their task produce 87.9–96.0% of all false rejections. Such a verifier gets a perfect negative-rejection rate on its task and never accepts a correct solution, so a high TNR alone hides it.

Analysis of Generated Verifiers

Tracing the shared checks back to generated code and generation-time self-tests shows three recurring mechanisms.

Reasoning effort spends the testing budget

Exploration and self-testing share a budget of 20 public environment executions per task. Going from LOW to HIGH effort, runs that hit a blocked environment request rise from 4 to 77 of 127 state and code tasks under the pi harness, and from 21 to 98 under Claude Code. Higher reasoning effort spends the generation budget on exploration and leaves the final verifier without a complete positive test.

Trajectory Cases

Five generation trajectories from the paper's appendix, each traced from the task requirement through the agent's own checks and self-tests to the hidden judgments. Blue panels are supplied requirements or agent actions, teal panels are observed execution feedback, and purple panels are hidden evaluation. Step through a case to follow it in order.

The cases explain particular observed failures and one successful paired configuration; they do not estimate how common each mechanism is. Record numbers refer to lines in the decompressed generation trace.

Implications

Ground every rejection condition. A rule that rejects should follow from the instruction, tool definitions, or environment rules, not from one implementation strategy.

Test whether a condition is necessary. Self-tests should vary an allowed property while keeping the task satisfied, and separately introduce a real violation.

Reserve budget for the final verifier. Interpret feedback by what was actually tested. A budget error is not evidence of a violation.

We do not evaluate the effectiveness of these directions.

Limitations

Scores measure agreement with retained references on finite, constructed test suites. The task-level and trace analysis covers one model under four conditions, inside a comparison of 36 configurations whose harnesses differ in prompts, tools, and reasoning settings. We do not evaluate the proposed improvements, downstream training gains, or multi-round self-improvement.

Citation

@inproceedings{verifierbench2026,
  title     = {VerifierBench: Benchmarking the Generation of
               Executable Verifiers for LLM Agents},
  author    = {Anonymous},
  booktitle = {Under review},
  year      = {2026}
}