SWE-bench, AgentBench, GAIA, and TAU-bench are the canonical agent benchmarks. List what each measures, the specific blind spots they share, and what a sound reading of agent benchmark numbers must include. Cover what changes between benchmark versions, why model version and harness version matter, why cost blind leaderboards are misleading, and what production teams should do beyond reading the leaderboard.
SWE-bench, AgentBench, GAIA, TAU-bench each measure something different; all share closed-loop, contamination, static distribution, and cost-blind blind spots. Quote model plus harness plus version.
Imagine four different driving tests. One tests parallel parking, one tests highway merging, one tests night driving, one tests city traffic with pedestrians. Each test gives you a score, but none of them captures all of driving. If someone tells you a driver scored 80 percent on one test, you still do not know if they can handle the others. Worse, if everyone has been studying the parallel-parking test for years, scoring well on it does not mean they have actually gotten better at parking unfamiliar cars in unfamiliar streets. And finally, if one driver passes the test by burning ten times as much fuel as another driver who passes the same test, the two drivers are not really equally good. Agent benchmarks are the same: useful as leading signals, dangerous as the only signal.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Agent benchmarks are a richer landscape than the single-shot QA benchmarks of the previous LLM generation. The four canonical benchmarks (SWE-bench, AgentBench, GAIA, TAU-bench) each measure something different and each has known limitations. The useful question is not which benchmark is best, but how to interpret any benchmark number well enough to make a deployment decision from it.
Benchmarks are calibration instruments, not deployment signals. They tell you a model can do a class of task in a controlled setting. They do not tell you it will perform on your specific workload. The blind spots are predictable and shared across the four benchmarks: closed loop scoring, recency contamination, static distributions, and cost blind leaderboards. The interpretation discipline is to always quote the triple (benchmark variant, model version, harness version), report accuracy alongside cost and latency, and run your own representative task eval before any deployment decision.
What each of the four benchmarks measures
SWE-bench scores software engineering trajectories on real GitHub PRs. The agent receives an issue description and the relevant repository, edits one or more files to address the issue, and the resulting patch is scored by running the held out test suite that the issue's PR also passed. A pass means the patch correctly resolves the issue without breaking other tests.
There are three variants. SWE-bench Verified is the 500-task subset that human reviewers validated as having reliable correctness signals and clear issue descriptions. This is the production relevant variant. SWE-bench Lite is a 300-task subset filtered for simple, self-contained fixes. SWE-bench Full is the original 2,294-task release with less curation. Quoting a SWE-bench number without specifying the variant is uninterpretable, because the variants use different filters and scores are comparable only within one variant.
AgentBench tests broad tool-use across 8 environments: SQL on databases, OS shell commands, web shopping, knowledge graphs, lateral thinking puzzles, web browsing, digital card games, and a household cleaning environment. Each environment has its own scoring, and the aggregate is a normalised mean. AgentBench captures generality in a way SWE-bench does not, but each environment is shallow relative to its real world counterpart, so per environment breakdowns are more useful than the aggregate.
GAIA is the general assistant multi-step reasoning benchmark. Tasks require web search, tool composition, multi-step reading, and factual reasoning. Three difficulty levels (1, 2, 3) roughly correspond to step count and complexity. GAIA is harder than it looks because the multi-step composition catches agents that ace simpler benchmarks; a model can be strong on individual steps and weak on planning across many of them.
TAU-bench is the customer service dialog benchmark. The agent talks to a simulated user, follows domain rules (booking, refunds, customer info), and calls tools to complete transactions. A task counts as solved when the final database state (and any required outputs) matches the annotated goal, and the benchmark reports pass^k over repeated trials. Policy violations show up only if they change the end state. TAU-bench is the closest of the four to a realistic production agent because it tests the full dialog plus tools loop with real conversational dynamics.
The four are not interchangeable 'agent benchmarks'. They cover different capability surfaces, and a model can be strong on one and weak on another in revealing ways.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- SWE-bench Verified launched in 2024 with 500 human validated tasks and is a common headline coding benchmark in model launches, though contamination concerns have pushed some labs toward newer sets such as SWE-bench Pro.
- TAU-bench is the production relevant dialog benchmark for customer service agents: it scores a task as solved when the final database state matches the annotated goal, and reports pass^k over repeated trials.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you actually run an internal representative task eval without overfitting to it?
Sample 200 to 500 tasks from production traffic with stratification on task type, difficulty, and customer segment. Label with ground truth where possible; use rubric based LLM-as-judge where not. Hold out a true test split that engineers never see during iteration. Rotate the held-out set periodically as the production distribution shifts. Track confidence intervals on every reported number to avoid mistaking noise for signal.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Quoting a benchmark percentage without specifying model version, harness version, and benchmark variant. SWE-bench scores from different harnesses are not directly comparable.
60 second bullets to scan on the way to the call.
Name SWE-bench, AgentBench, GAIA, and TAU-bench with their respective task domains.
Identify SWE-bench Verified, Lite, and Full as distinct variants with different difficulty.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.