🎯Today's Daily Challenge, completing this counts toward your streak.Today's set →

Match each agent benchmark to what it primarily measures

Match pairs·Medium·4.0 · 0·~2 min·Asked atBaseten Cursor Uber·Relevant atAnthropic

Attempt it

Drag each answer to line up with its matching prompt

SWE-bench

Multi-environment task completion across web, OS, and database tasks, measuring operational breadth

AgentBench

Realistic web navigation tasks on simulated live websites, measuring browser agent capability

WebArena

Autonomous GitHub issue resolution: produce a patch that makes a real test suite pass, measuring code editing depth

Agent benchmarks each test a different skill, so you cannot rank agents with a single number. The trick is to match each benchmark to the one thing it isolates. **SWE-bench** is the depth test. It hands the agent a real GitHub issue from a popular Python project plus the whole repository. The agent must write a code patch, and it only scores if the project's actual test suite passes afterward. There is no partial credit for looking busy. This measures whether an agent can do deep, verifiable software engineering. AgentBench is the breadth test. It runs the same agent across many different environments, including a web browser, an operating system shell, and a database. Each environment has its own small tasks. A high score means the agent is a generalist that adapts to varied tools, not a one-trick specialist. WebArena is the web navigation test. It puts the agent on simulated but lifelike websites, like a shopping site or a forum, and asks it to complete realistic goals by clicking, typing, and reading pages. This measures whether an agent can operate a browser the way a person does. The lesson: a strong SWE-bench score tells you nothing about web browsing, and vice versa. Always ask which axis a benchmark measures before trusting the number.

Agent benchmarks split along the axis each one isolates, so the match is about depth versus breadth versus a specific skill. **SWE-bench** measures code-editing depth. It gives the agent a real GitHub issue and the full repository, and the agent must produce a patch that makes the project's hidden test suite pass. Scoring is binary and verifiable, which is why it became the headline coding benchmark. AgentBench measures operational breadth. The same agent is run across eight environments spanning a shell, a database, web browsing, and more. A high score signals a generalist that handles varied tools rather than one narrow task. WebArena measures browser agent capability. It hosts self-contained, lifelike websites such as shopping and forum apps, then asks the agent to complete realistic multi-step goals through navigation and form filling. Two related benchmarks often come up. GAIA tests general assistant reasoning on questions that need tools, multi-step thinking, and synthesis. Humans score high on it, but agents historically struggle. tau-bench tests tool-using agents against a simulated user in customer-service domains. The unifying point is that no single score ranks agents, because each benchmark probes a different competence.

TL;DR

SWE-bench measures code-editing depth on real GitHub issues, AgentBench measures breadth across many environments, and WebArena measures realistic web navigation.

Memory aid

Easy to grasp

Imagine you want to test how good a new employee is. One test hands them a broken piece of software and a list of failing checks, and says fix it so the checks pass. That is SWE-bench, and it measures depth in one hard skill: writing real code. Another test sends them through many different rooms, a kitchen, a workshop, an office, and gives each room a small job. That is AgentBench, and it measures breadth across many situations. A third test drops them onto a fake but lifelike website and asks them to book a flight or find a product. That is WebArena, and it measures whether they can click around the web like a person. Each test asks a different question, so a worker who aces one can still flop on another.

Key concepts

Agent Evaluation SWE-bench AgentBench

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Agent benchmarks are easy to confuse because every one of them reports a percentage and every percentage looks like an overall grade. The interview skill is to resist that framing. Each benchmark deliberately isolates a different competence, and a strong number on one says almost nothing about the others. An agent that resolves seventy percent of coding issues may navigate a shopping site no better than chance, because the underlying skills barely overlap.

The clean mental model is a matrix of axes rather than a single leaderboard. SWE-bench is depth on real code. AgentBench is breadth across many environments. WebArena is browser grounding. GAIA is general assistant reasoning. tau-bench is tool use under policy with a simulated user. Once you map each name to its axis, the match-pairs question becomes mechanical, and the senior follow-ups about contamination and benchmark trust open up naturally.

The reason the field built so many benchmarks is that agents are systems, not models. A static question-answering dataset can grade a model in isolation, but an agent only exists when an LLM is wrapped in a loop with tools, state, and an environment. You cannot test that with a multiple-choice quiz. You have to put the whole system into a task, let it act over many steps, and check whether the world ended up in the goal state. Each benchmark below is a different world with a different goal, and that is exactly why they cannot be collapsed into one ranking.

SWE-bench: verifiable code-editing depth

SWE-bench is the depth benchmark for autonomous software engineering. Each task is a real issue pulled from a popular open source Python repository, such as Django or scikit-learn, paired with the full codebase as it existed at that moment. The agent must produce a patch that resolves the issue. This is genuinely hard, because the agent has to locate the relevant code across a large repository, understand the intended behaviour, and edit it without breaking anything else.

What makes it strict is the scoring. The patch is applied and the project's own hidden test suite is run, including tests written specifically for that issue that the agent never sees. The task counts as solved only if the relevant tests pass and no previously passing tests break. There is no credit for plausible-looking diffs, which is why the benchmark resists the reward hacking that plagues judge-based scores. A model cannot talk its way to a high number; the code either works or it does not.

The original release had problems: some issues were ambiguous, some were effectively unsolvable from the issue text alone, and some appeared in model training data because the repositories are public. The community responded with SWE-bench Verified, a human-filtered subset where engineers confirmed each task is well specified and solvable. That Verified number is the one most frontier launches now quote, and knowing the distinction is a quick way to show benchmark literacy. There are also harder variants such as SWE-bench Multimodal and live, continuously refreshed sets that fight contamination by using issues filed after a model's training cutoff.

AgentBench: breadth across environments

WebArena, GAIA, and tau-bench: three more axes

Why one score never ranks agents

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Benchmark	What it measures	Task type	Scoring signal
SWE-bench	Code-editing depth	Resolve a real GitHub issue	Hidden test suite passes
AgentBench	Operational breadth	Tasks across 8 environments	Per-environment success rate
WebArena	Browser navigation	Goals on self-hosted websites	Functional task completion
GAIA	General assistant reasoning	Multi-step questions needing tools	Exact-match answer correctness
tau-bench	Tool use under policy	Customer-service dialogue	Goal met plus policy followed

Real products, models, and research that use this idea.

SWE-bench Verified is the headline coding metric in frontier model launches, with GPT-5.5, Claude Opus 4.7, and Gemini 3.1 Pro all reporting resolution rates on it.
Cognition's Devin was first marketed on a SWE-bench resolution percentage, which set the template for agent coding claims.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy did the community shift from the original SWE-bench to SWE-bench Verified, and what does that say about benchmark trust?

Discuss training-data contamination and ambiguous or unsolvable issues in the original set. Verified is a human-filtered subset that removes those, raising confidence that a score reflects real capability rather than memorisation.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating all agent benchmarks as interchangeable scores. Each one isolates a different axis, so a high SWE-bench number says nothing about web navigation or broad task coverage.

60 second bullets to scan on the way to the call.

State what SWE-bench measures and why its scoring is binary.
Explain why AgentBench is described as a breadth benchmark.

Primary sources. Browse if you want the original framing.