Design a diagnostic eval for an agentic workflow with 62% task success across 5 tools and 10+ steps that reveals WHY tasks fail, not just whether they pass.

Short answer·Hard·4.0 · 0·~3 min·Relevant atDatabricks

Attempt it

Free · 2 AI evals / day

A 62% success rate on a multi-step agent workflow does not explain why 38% of tasks fail. The eval needs to go deeper. **Step-level scoring** checks each step independently. Was the right tool called? Did the arguments match the schema? Was the result interpreted correctly? Deterministic checks catch mechanical errors (wrong tool name, missing arguments). LLM-as-judge catches reasoning errors (wrong tool for the sub-task, bad argument values). **Trajectory efficiency** compares the agent's step count to an oracle (minimum-step solution). Extra steps reveal planning failures, loops, and redundant calls. **Error attribution** traces backward from a failed outcome to the first wrong step. Each root cause goes into a failure taxonomy: wrong tool selection (roughly 33%), correct tool with wrong arguments (25%), misinterpreted result (20%), hallucinated tool (12%), infrastructure timeout (10%). The taxonomy tells engineering where to invest: if one-third of failures are wrong tool selection, improve tool descriptions and add few-shot examples. Two diagnostic views complete the picture: per-tool success rate (which tools are problematic?) and per step position failure rate (where in the trajectory do errors concentrate?).

**Three scoring layers.** (1) Step-level correctness: deterministic checks (tool name exists, arguments match schema) plus LLM-as-judge semantic checks (right tool for the sub-task, reasonable argument values). (2) Trajectory efficiency: compare agent step count to oracle trajectory, flag redundant steps, loops, and unnecessary calls. (3) Error attribution: trace backward from failed outcome to the first wrong step. **Failure taxonomy.** Wrong tool selection (roughly 33%): fix with better tool descriptions and few-shot examples. Correct tool, wrong arguments (25%): fix with better documentation and schema validation. Misinterpreted result (20%): fix with structured output parsing. Hallucinated nonexistent tool (12%): fix with tool-name validation and prominent tool-set listing. Infrastructure timeout (10%): fix with retry logic and fallback paths. **Diagnostic views.** Per-tool success rate reveals which tools the agent struggles with. Per-step position failure rate reveals where in the trajectory errors concentrate. If step 3 has a 25% failure rate while other positions are below 10%, the agent has a consistent planning breakdown at that trajectory depth. **Existing patterns.** SWE-bench Verified, TAU-bench, and AgentBench already use trajectory scoring. Adapt their rubric patterns to the specific tool set. The taxonomy drives engineering priorities directly: percentages map to which fixes yield the highest return.

Quick reference

TL;DR

Three scoring layers (step-level correctness, trajectory efficiency, backward error attribution) plus a failure taxonomy that tells engineering exactly where to invest to move the 62% success rate.

Memory aid

Easy to grasp

Imagine a cooking competition where contestants follow a 10-step recipe using 5 different kitchen tools. If 38% of dishes taste bad, the judges need to know why. Did the chef pick the wrong tool (used a blender instead of a mixer)? Did they use the right tool with wrong settings (oven at 500 degrees instead of 350)? Did they misread the timer? By watching each step and noting where things first went wrong, you can tell the chef exactly what to practice instead of just saying the dish failed.

Key concepts

Agent Evaluation SWE-bench

Detailed answer & concept explanation

~9 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

Open by framing the problem: 62% pass rate, PM wants to know why 38% fail. Walk the three scoring layers in order: step-level correctness (deterministic plus semantic), trajectory efficiency (oracle comparison), error attribution (backward trace). Present the failure taxonomy with approximate percentages and map each category to a specific engineering fix. Describe the two diagnostic views (per-tool, per step position). Reference SWE-bench, TAU-bench, AgentBench as proof that trajectory scoring works at scale.

Real-world examples

Real products, models, and research that use this idea.

SWE-bench Verified evaluates software-engineering agent trajectories with per-step analysis of code edits, using test suite execution as the deterministic correctness check and trajectory length as an efficiency metric.
TAU-bench scores customer-service agent dialog per turn, checking rule-following and tool-call validity at each step, directly implementing the step-level correctness layer described here.
AgentBench evaluates agents across 8 environments with per-environment trajectory analysis, demonstrating that the three-layer scoring pattern scales across diverse tool sets.
Anthropic's internal agent evaluation pipeline uses backward error attribution to trace task failures to root-cause steps, with a failure taxonomy that directly informs agent improvement priorities.
Production agentic workflows at scale in 2026 commonly instrument per-tool success rates and per step position failure rates as the primary diagnostic signals for agent quality improvement.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you build the oracle trajectory for tasks where the optimal solution is not obvious?

QThe agent calls tool B correctly at step 4 but then misinterprets the result at step 5. How does the LLM-as-judge score step 5?

QThe failure taxonomy shows 33% wrong tool selection. How do you determine whether to fix tool descriptions versus reducing the total tool count?

QHow would you handle the case where the agent fails at step 3 but recovers at step 7 and ultimately succeeds?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Scoring only the final task outcome and reporting a single pass/fail number. This tells the PM that 38% fail but not whether the failures are concentrated in one tool, one step position, or one error type, making it impossible to prioritize fixes.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.