How should you score a multi-step agent where final answer accuracy is 15% but most steps are correct?

MCQ·Medium·4.0 · 0·~1 min·Asked atCapgemini Databricks Decagon

Attempt it

When the final answer is rare (15 percent) but most steps along the way are correct (60 percent), neither metric tells you what is happening on its own. Final-answer accuracy is **too sparse**. A shift from 15 percent to 17 percent looks like noise unless you run hundreds of tasks. Day to day regressions get lost in the wobble. Step accuracy is **too generous**. An agent that does 50 mediocre steps without ever finishing scores well, even though it is useless. The right design is a **composite metric**. You combine four signals: final answer accuracy (does the task actually complete correctly), per step correctness (did each intermediate step do the right thing), tool-call validity (did the agent call tools with valid schemas and arguments), and completion rate (did the agent finish or stall). The weights put most emphasis on the final answer, because that is what users care about, but the other signals give you the intermediate visibility you need to debug. On top of the composite, you tag each failing task with a **failure mode bucket**: wrong tool, wrong args, wrong reasoning, gave up, infinite loop. Now your eval is not just a number, it is a breakdown that tells you what to fix. SWE-bench, AgentBench, TAU-bench, and the rest of the canonical agent benchmarks all use this style of composite scoring with per-task tagging.

Use a **weighted composite**: final answer accuracy as the primary signal, per step correctness as the intermediate signal, tool-call validity as the schema level signal, and completion rate as the stall signal. Weight final answer most heavily because that is the user visible outcome, but the other signals are what make the metric debuggable. Pure final answer is too sparse at 15 percent baseline. A 2-point shift is statistically indistinguishable from noise without a large eval set, so regressions hide. Pure step accuracy rewards thrashing: an agent that emits many small correct steps without ever finishing looks fine. The composite catches both failure modes. Add **failure mode tagging** per task. Each failing run gets bucketed: wrong tool choice, wrong tool arguments, wrong reasoning, gave up, infinite loop, broken output. The eval report shows both the headline composite and the bucket distribution. A change that improves the composite by 3 points but doubles the gave up bucket is a regression in disguise. SWE-bench, AgentBench, GAIA, and TAU-bench all use composite scoring for the same reason. SWE-bench, for example, requires the agent's diff to pass the held out test suite, which is a composite of multiple test outcomes per task, not a single pass or fail on one assertion. Production tracking adds two things on top: trend lines over time so you see whether failure mode distribution is shifting, and per-tenant or per task type slicing so you can detect that a specific class of task is regressing while the headline looks stable.

TL;DR

A weighted composite of final answer, per step correctness, tool-call validity, and completion rate, plus failure mode tagging, gives actionable signal that neither metric alone provides.

Memory aid

Easy to grasp

Imagine grading a student on a 10-question test where they almost always get the last hard question wrong but get most of the rest right. If you grade only on the last question, you cannot tell if they are improving on the easy ones, because they almost never get the last one right. If you grade only on the average across all questions, a student who answers 8 right and skips 2 looks worse than one who fakes 10 mediocre answers. So you use a scorecard: most of the grade comes from the last question (it is the goal), but partial credit comes from the others. And you write a short note on each test saying what kind of mistake the student made, so you can tell over time whether their wrong answers are getting better or worse.

Key concepts

Agent Evaluation Composite Metrics SWE-bench

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Multi-step agents create a specific evaluation problem: the outcome you ultimately care about (did the task succeed?) is rare, and the signal you have along the way (how many steps were correct?) is plentiful. The question's numbers, 15 percent final answer accuracy and 60 percent average step accuracy, are realistic for a complex production agent and create a real trap. Either metric used alone gives misleading signal.

The correct answer is a weighted composite of final answer accuracy, per step correctness, tool-call validity, and completion rate, with per-task failure mode tagging on top. The canonical 2026 agent benchmarks (SWE-bench, AgentBench, GAIA, TAU-bench) all use this style for the same reason: the metric needs to be both dense enough to detect regressions in iteration and faithful enough to predict user facing outcomes.

The rest of this explanation walks through why each single metric approach fails, how the composite is constructed and weighted, why failure mode tagging is the load bearing addition that makes the eval actionable, and the production grade nuances around cost reporting, statistical significance, and LLM-as-judge calibration that separate a usable eval from a misleading one.

Why pure final answer accuracy fails at 15 percent

Final-answer accuracy is the metric that matches what users see, so it is tempting to make it the only metric. The problem at a 15 percent base rate is statistical, not philosophical: the metric is too sparse to drive day to day iteration.

Consider the variance. On a 100-task eval set, a true accuracy of 15 percent has a standard error of roughly 3.6 points. A measured improvement from 15 percent to 18 percent is well within one standard error and statistically indistinguishable from luck. To detect a 3-point real improvement reliably, you need an eval set in the thousands of tasks, which is expensive to run and expensive to maintain. Smaller meaningful changes (1 to 2 points) need even larger sets.

The practical consequence is that final answer only eval cannot guide week to week iteration. Engineers ship a change, run the eval, see a noisy result, and have no way to tell whether the change helped or hurt. The natural human response is to overfit to whichever direction the noise pushed the metric, which produces a release where the eval drifts on noise rather than on real signal.

A secondary issue is the lack of debuggability. When an agent fails on a task, you want to know what kind of failure: did it pick the wrong tool, call the right tool with bad arguments, reason incorrectly, give up, get stuck in a loop? Final-answer accuracy collapses all of these into one bit, which makes it impossible to tell from the metric alone what to fix.

The deeper lesson is that any metric with a low base rate is starved of signal. The fix is not to give up on it but to combine it with denser intermediate signals that, together, give you both the user facing outcome and the engineering visibility.

Why pure step accuracy rewards thrashing

Designing the composite: four channels and weights

Failure-mode tagging: turning the metric into an engineering tool

Production nuances: cost, significance, slicing, and judge drift

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

SWE-bench scores agents by whether their patch makes the held out test suite pass on real GitHub issues; Verified, Lite, and Full variants use the same composite pass test but on different task subsets.
AgentBench evaluates across 8 different tool-use environments and reports both per-environment scores and an aggregate, treating the suite as a composite rather than a single number.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you choose the weights on the composite without making the choice arbitrary?

Anchor to the user facing outcome. The final answer weight should be high enough that an agent improving only intermediate signals without improving the final answer does not look better on the headline. A common heuristic is to set the weights so the composite ranking matches the human ranking on a small judgment set, then validate that the ranking is stable across eval slices.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Tracking only final answer accuracy, which is too sparse at 15 percent to detect regressions, or only step accuracy, which rewards an agent that thrashes through many small correct steps without finishing.

60 second bullets to scan on the way to the call.

Explain why a 15 percent final answer rate is too sparse for day to day eval signal.
Identify the failure mode of pure step accuracy: rewarding thrashing without completion.

Primary sources. Browse if you want the original framing.