Agent benchmarks each test a different skill, so you cannot rank agents with a single number. The trick is to match each benchmark to the one thing it isolates.
**SWE-bench** is the depth test. It hands the agent a real GitHub issue from a popular Python project plus the whole repository. The agent must write a code patch, and it only scores if the project's actual test suite passes afterward. There is no partial credit for looking busy. This measures whether an agent can do deep, verifiable software engineering.
AgentBench is the breadth test. It runs the same agent across many different environments, including a web browser, an operating system shell, and a database. Each environment has its own small tasks. A high score means the agent is a generalist that adapts to varied tools, not a one-trick specialist.
WebArena is the web navigation test. It puts the agent on simulated but lifelike websites, like a shopping site or a forum, and asks it to complete realistic goals by clicking, typing, and reading pages. This measures whether an agent can operate a browser the way a person does.
The lesson: a strong SWE-bench score tells you nothing about web browsing, and vice versa. Always ask which axis a benchmark measures before trusting the number.
Agent benchmarks split along the axis each one isolates, so the match is about depth versus breadth versus a specific skill.
**SWE-bench** measures code-editing depth. It gives the agent a real GitHub issue and the full repository, and the agent must produce a patch that makes the project's hidden test suite pass. Scoring is binary and verifiable, which is why it became the headline coding benchmark.
AgentBench measures operational breadth. The same agent is run across eight environments spanning a shell, a database, web browsing, and more. A high score signals a generalist that handles varied tools rather than one narrow task.
WebArena measures browser agent capability. It hosts self-contained, lifelike websites such as shopping and forum apps, then asks the agent to complete realistic multi-step goals through navigation and form filling.
Two related benchmarks often come up. GAIA tests general assistant reasoning on questions that need tools, multi-step thinking, and synthesis. Humans score high on it, but agents historically struggle. tau-bench tests tool-using agents against a simulated user in customer-service domains. The unifying point is that no single score ranks agents, because each benchmark probes a different competence.
Quick reference
TL;DR
SWE-bench measures code-editing depth on real GitHub issues, AgentBench measures breadth across many environments, and WebArena measures realistic web navigation.
Imagine you want to test how good a new employee is. One test hands them a broken piece of software and a list of failing checks, and says fix it so the checks pass. That is SWE-bench, and it measures depth in one hard skill: writing real code. Another test sends them through many different rooms, a kitchen, a workshop, an office, and gives each room a small job. That is AgentBench, and it measures breadth across many situations. A third test drops them onto a fake but lifelike website and asks them to book a flight or find a product. That is WebArena, and it measures whether they can click around the web like a person. Each test asks a different question, so a worker who aces one can still flop on another.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
You need to judge an agent that only operates inside a private internal tool stack. Public benchmarks test different environments, so they will not reflect your domain-specific success.
You want a single headline score to rank vendors for a procurement decision. Each benchmark isolates one axis, so one number hides where an agent is weak.
Your task is pure single-shot retrieval with no multi-step tool use at all. Agent benchmarks assume iterative tool calling, so they over-test capabilities you will never exercise.
You are evaluating safety, cost, or latency rather than task success. These benchmarks score task completion only and ignore the operational properties you actually care about.
Quick scan · 2–4 min
Everything important, quickly.
Match each benchmark to its axis first, depth for SWE-bench, breadth for AgentBench, browser for WebArena, then add GAIA for general reasoning and tau-bench for tool use under a simulated user, and close on why a single score cannot rank agents and why SWE-bench Verified is the trusted number.
Side-by-side
Benchmark
What it measures
Task type
Scoring signal
SWE-bench
Code-editing depth
Resolve a real GitHub issue
Hidden test suite passes
AgentBench
Operational breadth
Tasks across 8 environments
Per-environment success rate
WebArena
Browser navigation
Goals on self-hosted websites
Functional task completion
GAIA
General assistant reasoning
Multi-step questions needing tools
Exact-match answer correctness
tau-bench
Tool use under policy
Customer-service dialogue
Goal met plus policy followed
Real-world examples
Real products, models, and research that use this idea.
1SWE-bench Verified is the headline coding metric in frontier model launches, with GPT-5.5, Claude Opus 4.7, and Gemini 3.1 Pro all reporting resolution rates on it.
2Cognition's Devin was first marketed on a SWE-bench resolution percentage, which set the template for agent coding claims.
3GAIA is used by assistant-style products to show how far tool-using agents are from the near-perfect human baseline on multi-step questions.
4tau-bench is cited by teams building customer-service agents to prove the agent follows policy across a simulated user conversation, not just a single turn.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy did the community shift from the original SWE-bench to SWE-bench Verified, and what does that say about benchmark trust?
Sign in or go Pro to see the approach for this follow-up.
QHow would you design an evaluation suite for a production coding agent that goes beyond SWE-bench?
Sign in or go Pro to see the approach for this follow-up.
QWhat makes WebArena scores hard to compare across labs even on the same tasks?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating all agent benchmarks as interchangeable scores. Each one isolates a different axis, so a high SWE-bench number says nothing about web navigation or broad task coverage.