How should you score a multi-step agent where final answer accuracy is 15% but most steps are correct?

MCQ·Medium·4.0 · 0·~1 min·Asked atDatabricks

Attempt it

When the final answer is rare (15 percent) but most steps along the way are correct (60 percent), neither metric tells you what is happening on its own. Final-answer accuracy is **too sparse**. A shift from 15 percent to 17 percent looks like noise unless you run hundreds of tasks. Day to day regressions get lost in the wobble. Step accuracy is **too generous**. An agent that does 50 mediocre steps without ever finishing scores well, even though it is useless. The right design is a **composite metric**. You combine four signals: final answer accuracy (does the task actually complete correctly), per step correctness (did each intermediate step do the right thing), tool-call validity (did the agent call tools with valid schemas and arguments), and completion rate (did the agent finish or stall). The weights put most emphasis on the final answer, because that is what users care about, but the other signals give you the intermediate visibility you need to debug. On top of the composite, you tag each failing task with a **failure mode bucket**: wrong tool, wrong args, wrong reasoning, gave up, infinite loop. Now your eval is not just a number, it is a breakdown that tells you what to fix. SWE-bench, AgentBench, TAU-bench, and the rest of the canonical agent benchmarks all use this style of composite scoring with per-task tagging.

Use a **weighted composite**: final answer accuracy as the primary signal, per step correctness as the intermediate signal, tool-call validity as the schema level signal, and completion rate as the stall signal. Weight final answer most heavily because that is the user visible outcome, but the other signals are what make the metric debuggable. Pure final answer is too sparse at 15 percent baseline. A 2-point shift is statistically indistinguishable from noise without a large eval set, so regressions hide. Pure step accuracy rewards thrashing: an agent that emits many small correct steps without ever finishing looks fine. The composite catches both failure modes. Add **failure mode tagging** per task. Each failing run gets bucketed: wrong tool choice, wrong tool arguments, wrong reasoning, gave up, infinite loop, broken output. The eval report shows both the headline composite and the bucket distribution. A change that improves the composite by 3 points but doubles the gave up bucket is a regression in disguise. SWE-bench, AgentBench, GAIA, and TAU-bench all use composite scoring for the same reason. SWE-bench, for example, requires the agent's diff to pass the held out test suite, which is a composite of multiple test outcomes per task, not a single pass or fail on one assertion. Production tracking adds two things on top: trend lines over time so you see whether failure mode distribution is shifting, and per-tenant or per task type slicing so you can detect that a specific class of task is regressing while the headline looks stable.

Quick reference

TL;DR

A weighted composite of final answer, per step correctness, tool-call validity, and completion rate, plus failure mode tagging, gives actionable signal that neither metric alone provides.

Memory aid

Easy to grasp

Imagine grading a student on a 10-question test where they almost always get the last hard question wrong but get most of the rest right. If you grade only on the last question, you cannot tell if they are improving on the easy ones, because they almost never get the last one right. If you grade only on the average across all questions, a student who answers 8 right and skips 2 looks worse than one who fakes 10 mediocre answers. So you use a scorecard: most of the grade comes from the last question (it is the goal), but partial credit comes from the others. And you write a short note on each test saying what kind of mistake the student made, so you can tell over time whether their wrong answers are getting better or worse.

Key concepts

Agent Evaluation Composite Metrics SWE-bench

Detailed answer & concept explanation

~9 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

Open with the diagnostic: sparse final answer plus dense intermediate signal forces a composite. Name the four channels and weights, explain why pure final answer and pure step accuracy each fail. Cover failure mode tagging as what makes the eval actionable. Anchor to SWE-bench, AgentBench, GAIA, and TAU-bench as canonical examples. Close on cost per task, confidence intervals, and LLM-as-judge calibration as production grade nuances.

Real-world examples

Real products, models, and research that use this idea.

SWE-bench scores agents by whether their patch makes the held out test suite pass on real GitHub issues; Verified, Lite, and Full variants use the same composite pass test but on different task subsets.
AgentBench evaluates across 8 different tool-use environments and reports both per-environment scores and an aggregate, treating the suite as a composite rather than a single number.
TAU-bench measures customer service dialog agents with rule following plus task completion plus tool-call correctness, all combined into the headline pass rate.
Anthropic's evaluation docs explicitly recommend composite scoring with failure mode tagging for multi-step agent tasks, citing the same sparse-outcome problem the question describes.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you choose the weights on the composite without making the choice arbitrary?

QHow do you handle tasks where there is no single ground truth final answer, such as an open ended research task?

QYour eval composite improves but production user satisfaction stays flat. What did the metric miss?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Tracking only final answer accuracy, which is too sparse at 15 percent to detect regressions, or only step accuracy, which rewards an agent that thrashes through many small correct steps without finishing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.