What does SWE-bench measure that AgentBench does not, and why are both needed?

Agent benchmarks each test a different slice of what an agent can do, and no single one tells the whole story. **SWE-bench** gives the agent a real GitHub issue plus the repository, and asks it to produce a patch. The patch only counts if the project's existing test suite passes. This is a deep test of code-editing skill, and grading is automatic and honest because tests either pass or fail. AgentBench is different. It throws the agent into many environments: a web browser, an operating-system shell, a database, a card game. It measures breadth, how well one agent copes across very different tool types rather than one specialty. Two more benchmarks fill remaining gaps. GAIA asks general-assistant questions that need real reasoning and tool use, with one verifiable answer per question. Tau-bench simulates multi-turn customer-service conversations and checks whether the right outcome happened, like a refund actually issued. The point is that depth, breadth, reasoning, and multi-turn reliability are separate axes. A great coder can flop at web navigation, so you read several benchmarks together instead of trusting one number.

Each benchmark probes a different axis, and a single score never characterises an agent. **SWE-bench** tests deep coding. The agent gets a real GitHub issue and repo, and must produce a patch that makes the hidden test suite pass. Grading is execution-based, so it is objective. AgentBench tests breadth. It runs the agent across web, shell, database, and game environments to measure cross-domain generalisation. GAIA tests general-assistant reasoning with multi-step tool use and one verifiable answer per question. Tau-bench tests multi-turn reliability in simulated customer-service dialogue. It scores whether the correct end state was reached, not whether the prose sounded right. The execution-based versus judged split matters most. SWE-bench and tau-bench verify outcomes against ground truth. AgentBench and many leaderboards lean on model-graded or rubric scoring, which a confident wrong answer can game. Two failure modes break single-benchmark trust. Contamination means training data may include the test issues, inflating scores. Saturation means top models now cluster near the ceiling, so the benchmark stops discriminating. No benchmark captures real reliability, which is consistency across long, messy, unseen tasks. You triangulate across several axes instead.