Same topic, related formats. Practice these next.
Distinguish SWE-bench from AgentBench. What capability dimension does each benchmark cover, and why is performance on one insufficient to characterise an agent's full capability?
SWE-bench measures execution-verified coding depth; AgentBench measures multi-environment breadth. Neither captures real reliability, so you triangulate across both plus GAIA and tau-bench.
Imagine you want to know how good a new hire really is. One test makes them fix a real bug in a big codebase and only counts it if the tests pass afterward. That is a deep, honest test, but it only covers coding. Another test gives them many different small jobs: browse a website, run a few shell commands, query a database. That tells you how broadly they cope, but a judge grading the answers can be fooled by a confident wrong reply. A third test asks puzzle-like research questions, and a fourth watches how they handle a customer over many back and forth turns. Each test sees one slice of the person. If you only trust one, you get a misleading picture. Real ability shows up only when you line up several different tests and compare what they each reveal.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
Lay out the four axes first, depth, breadth, reasoning, and multi-turn reliability, then map SWE-bench, AgentBench, GAIA, and tau-bench onto them. Contrast execution-based against judged grading, raise contamination and saturation as validity threats, and close on why production reliability forces triangulation plus private held-out evals.
| Benchmark | Axis measured | Grading | Main weakness |
|---|---|---|---|
| SWE-bench | Code-editing depth on real repos | Execution-based (tests pass or fail) | Narrow to coding; contamination of public issues |
| AgentBench | Breadth across 8 environments | Mixed: some execution, some judged | Judged sub-tasks are gameable by fluent wrong answers |
| GAIA | General-assistant multi-step reasoning | Verifiable single answer per question | Saturating at the top; limited domain coverage |
| tau-bench | Multi-turn reliability in dialogue | Final-state outcome check | Simulated user may not reflect real customers |
Real products, models, and research that use this idea.
What an interviewer would ask next. Try answering before peeking at the approach.
Red flags and common mistakes that signal junior thinking. Click to expand.
Treating one headline benchmark score as the agent's capability. Each benchmark probes a narrow slice, and an execution-verified coding number says nothing about web navigation, multi-turn reliability, or judged task quality.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.