Match each agent benchmark to what it primarily measures

Agent benchmarks each test a different skill, so you cannot rank agents with a single number. The trick is to match each benchmark to the one thing it isolates. **SWE-bench** is the depth test. It hands the agent a real GitHub issue from a popular Python project plus the whole repository. The agent must write a code patch, and it only scores if the project's actual test suite passes afterward. There is no partial credit for looking busy. This measures whether an agent can do deep, verifiable software engineering. AgentBench is the breadth test. It runs the same agent across many different environments, including a web browser, an operating system shell, and a database. Each environment has its own small tasks. A high score means the agent is a generalist that adapts to varied tools, not a one-trick specialist. WebArena is the web navigation test. It puts the agent on simulated but lifelike websites, like a shopping site or a forum, and asks it to complete realistic goals by clicking, typing, and reading pages. This measures whether an agent can operate a browser the way a person does. The lesson: a strong SWE-bench score tells you nothing about web browsing, and vice versa. Always ask which axis a benchmark measures before trusting the number.

Agent benchmarks split along the axis each one isolates, so the match is about depth versus breadth versus a specific skill. **SWE-bench** measures code-editing depth. It gives the agent a real GitHub issue and the full repository, and the agent must produce a patch that makes the project's hidden test suite pass. Scoring is binary and verifiable, which is why it became the headline coding benchmark. AgentBench measures operational breadth. The same agent is run across eight environments spanning a shell, a database, web browsing, and more. A high score signals a generalist that handles varied tools rather than one narrow task. WebArena measures browser agent capability. It hosts self-contained, lifelike websites such as shopping and forum apps, then asks the agent to complete realistic multi-step goals through navigation and form filling. Two related benchmarks often come up. GAIA tests general assistant reasoning on questions that need tools, multi-step thinking, and synthesis. Humans score high on it, but agents historically struggle. tau-bench tests tool-using agents against a simulated user in customer-service domains. The unifying point is that no single score ranks agents, because each benchmark probes a different competence.