Distinguish SWE-bench from AgentBench. What capability dimension does each benchmark cover, and why is performance on one insufficient to characterise an agent's full capability?
SWE-bench measures execution-verified coding depth; AgentBench measures multi-environment breadth. Neither captures real reliability, so you triangulate across both plus GAIA and tau-bench.
Imagine you want to know how good a new hire really is. One test makes them fix a real bug in a big codebase and only counts it if the tests pass afterward. That is a deep, honest test, but it only covers coding. Another test gives them many different small jobs: browse a website, run a few shell commands, query a database. That tells you how broadly they cope, but a judge grading the answers can be fooled by a confident wrong reply. A third test asks puzzle-like research questions, and a fourth watches how they handle a customer over many back and forth turns. Each test sees one slice of the person. If you only trust one, you get a misleading picture. Real ability shows up only when you line up several different tests and compare what they each reveal.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Agent evaluation is not one number. It is a set of axes, and every public benchmark illuminates one of them while leaving the others dark. The question pairs SWE-bench and AgentBench, but the honest senior answer widens the frame to four benchmarks, because the moment you accept that depth and breadth are different axes, you also have to account for reasoning quality and multi-turn reliability.
The central tension is that a single headline score is almost always misleading. A coding agent can post a strong SWE-bench result and still flail on a shell task, a multi-step research query, or a forty-turn customer conversation. Worse, two structural problems, contamination and saturation, quietly erode the meaning of any one score over time. The skill being tested is whether you can name the axes, match benchmarks to them, and explain why you triangulate rather than trust one leaderboard.
SWE-bench: the depth axis, execution-verified
SWE-bench hands the agent a real GitHub issue and the corresponding repository at the commit just before the fix. The agent must produce a patch. The run counts as solved only if the repository's hidden test suite, the same tests the human fix had to satisfy, passes after the patch is applied. There is no rubric and no judge. Tests pass or they do not. The task instances are drawn from popular open-source Python projects like Django, scikit-learn, and matplotlib, so the agent is editing mature code with real conventions and real test coverage rather than toy snippets.
This makes SWE-bench an execution-based benchmark, which is its great strength. A fluent but wrong patch cannot talk its way past a failing assertion. It probes a genuinely deep skill stack: localising a bug from a natural-language report, navigating an unfamiliar codebase, making coordinated multi-file edits, and respecting existing interfaces. The agent often has to read several files, trace a call path, and reproduce the reported failure before it can even attempt a fix, which is exactly the workflow a human engineer follows.
The scoring distinguishes two checks that matter. Fail-to-pass tests are the ones that were broken before and must pass after; pass-to-pass tests guard against regressions the patch might introduce. An agent that fixes the reported bug but breaks an unrelated test does not get credit, which mirrors the real cost of a careless change in production code.
The weakness is that it is narrow. It measures coding and nothing else. A perfect SWE-bench agent tells you nothing about whether the same system can operate a browser or query a database. SWE-bench Verified was later curated, with human annotators removing instances that were unsolvable, underspecified, or had flaky or overly strict tests, tightening the signal but not widening the axis. Variants like SWE-bench Lite and SWE-bench Multimodal trade coverage for speed or add screenshots, but they all stay firmly on the coding axis.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Benchmark | Axis measured | Grading | Main weakness |
|---|---|---|---|
| SWE-bench | Code-editing depth on real repos | Execution-based (tests pass or fail) | Narrow to coding; contamination of public issues |
| AgentBench | Breadth across 8 environments | Mixed: some execution, some judged | Judged sub-tasks are gameable by fluent wrong answers |
| GAIA | General-assistant multi-step reasoning | Verifiable single answer per question | Saturating at the top; limited domain coverage |
| tau-bench | Multi-turn reliability in dialogue | Final-state outcome check | Simulated user may not reflect real customers |
Real products, models, and research that use this idea.
- Frontier labs report SWE-bench Verified resolution rates for coding agents; by 2026 GPT-5.5 and Claude Opus 4.7 cluster high enough that the original split barely discriminates between them.
- Tau-bench is used to evaluate customer-service agents by checking final database state, such as whether a refund or booking change was actually applied under policy.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you detect that a high public SWE-bench score is inflated by contamination rather than genuine capability?
Hold out a private set of recent issues created after the model's training cutoff, and compare resolution rate against the public split. A large gap, or near-zero performance on freshly authored issues with renamed identifiers, points to memorisation.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating one headline benchmark score as the agent's capability. Each benchmark probes a narrow slice, and an execution-verified coding number says nothing about web navigation, multi-turn reliability, or judged task quality.
60 second bullets to scan on the way to the call.
Describe what SWE-bench gives the agent and how a run is scored.
Explain what breadth means and which environments AgentBench spans.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.