Design a diagnostic eval for an agentic workflow with 62% task success across 5 tools and 10+ steps that reveals WHY tasks fail, not just whether they pass.
Short answer·Hard·4.0 · 0·~3 min·Relevant atDatabricks
Design a diagnostic eval for an agentic workflow with 62% task success across 5 tools and 10+ steps that reveals WHY tasks fail, not just whether they pass.
Free · 2 AI evals / day
A 62% success rate on a multi-step agent workflow does not explain why 38% of tasks fail. The eval needs to go deeper.
**Step-level scoring** checks each step independently. Was the right tool called? Did the arguments match the schema? Was the result interpreted correctly? Deterministic checks catch mechanical errors (wrong tool name, missing arguments). LLM-as-judge catches reasoning errors (wrong tool for the sub-task, bad argument values).
**Trajectory efficiency** compares the agent's step count to an oracle (minimum-step solution). Extra steps reveal planning failures, loops, and redundant calls.
**Error attribution** traces backward from a failed outcome to the first wrong step. Each root cause goes into a failure taxonomy: wrong tool selection (roughly 33%), correct tool with wrong arguments (25%), misinterpreted result (20%), hallucinated tool (12%), infrastructure timeout (10%). The taxonomy tells engineering where to invest: if one-third of failures are wrong tool selection, improve tool descriptions and add few-shot examples.
Two diagnostic views complete the picture: per-tool success rate (which tools are problematic?) and per step position failure rate (where in the trajectory do errors concentrate?).
**Three scoring layers.** (1) Step-level correctness: deterministic checks (tool name exists, arguments match schema) plus LLM-as-judge semantic checks (right tool for the sub-task, reasonable argument values). (2) Trajectory efficiency: compare agent step count to oracle trajectory, flag redundant steps, loops, and unnecessary calls. (3) Error attribution: trace backward from failed outcome to the first wrong step.
**Failure taxonomy.** Wrong tool selection (roughly 33%): fix with better tool descriptions and few-shot examples. Correct tool, wrong arguments (25%): fix with better documentation and schema validation. Misinterpreted result (20%): fix with structured output parsing. Hallucinated nonexistent tool (12%): fix with tool-name validation and prominent tool-set listing. Infrastructure timeout (10%): fix with retry logic and fallback paths.
**Diagnostic views.** Per-tool success rate reveals which tools the agent struggles with. Per-step position failure rate reveals where in the trajectory errors concentrate. If step 3 has a 25% failure rate while other positions are below 10%, the agent has a consistent planning breakdown at that trajectory depth.
**Existing patterns.** SWE-bench Verified, TAU-bench, and AgentBench already use trajectory scoring. Adapt their rubric patterns to the specific tool set. The taxonomy drives engineering priorities directly: percentages map to which fixes yield the highest return.
Quick reference
TL;DR
Three scoring layers (step-level correctness, trajectory efficiency, backward error attribution) plus a failure taxonomy that tells engineering exactly where to invest to move the 62% success rate.
Imagine a cooking competition where contestants follow a 10-step recipe using 5 different kitchen tools. If 38% of dishes taste bad, the judges need to know why. Did the chef pick the wrong tool (used a blender instead of a mixer)? Did they use the right tool with wrong settings (oven at 500 degrees instead of 350)? Did they misread the timer? By watching each step and noting where things first went wrong, you can tell the chef exactly what to practice instead of just saying the dish failed.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
A single-step LLM call with no tool use. Trajectory scoring requires a multi-step trajectory with tool calls. A single-step call has no trajectory to analyze. Standard output-quality evaluation is sufficient.
An agent with a fixed, deterministic workflow where the steps do not vary. If the agent always calls the same tools in the same order, trajectory-level scoring collapses to per-step output scoring. The efficiency and tool-selection layers add no diagnostic value.
An early prototype where the tool set is still changing weekly. The failure taxonomy and per-tool success rates are unstable when tools are added and removed frequently. Stabilize the tool set first, then invest in trajectory-level eval.
A task where only the final output matters and the path is irrelevant (e.g., mathematical proof generation). If the user does not care how the agent arrived at the answer, trajectory efficiency and step-level scoring add overhead without user-facing value. Score the final output directly.
Quick scan · 2–4 min
Everything important, quickly.
Open by framing the problem: 62% pass rate, PM wants to know why 38% fail. Walk the three scoring layers in order: step-level correctness (deterministic plus semantic), trajectory efficiency (oracle comparison), error attribution (backward trace). Present the failure taxonomy with approximate percentages and map each category to a specific engineering fix. Describe the two diagnostic views (per-tool, per step position). Reference SWE-bench, TAU-bench, AgentBench as proof that trajectory scoring works at scale.
Real-world examples
Real products, models, and research that use this idea.
1SWE-bench Verified evaluates software-engineering agent trajectories with per-step analysis of code edits, using test suite execution as the deterministic correctness check and trajectory length as an efficiency metric.
2TAU-bench scores customer-service agent dialog per turn, checking rule-following and tool-call validity at each step, directly implementing the step-level correctness layer described here.
3AgentBench evaluates agents across 8 environments with per-environment trajectory analysis, demonstrating that the three-layer scoring pattern scales across diverse tool sets.
4Anthropic's internal agent evaluation pipeline uses backward error attribution to trace task failures to root-cause steps, with a failure taxonomy that directly informs agent improvement priorities.
5Production agentic workflows at scale in 2026 commonly instrument per-tool success rates and per step position failure rates as the primary diagnostic signals for agent quality improvement.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you build the oracle trajectory for tasks where the optimal solution is not obvious?
Sign in or go Pro to see the approach for this follow-up.
QThe agent calls tool B correctly at step 4 but then misinterprets the result at step 5. How does the LLM-as-judge score step 5?
Sign in or go Pro to see the approach for this follow-up.
QThe failure taxonomy shows 33% wrong tool selection. How do you determine whether to fix tool descriptions versus reducing the total tool count?
Sign in or go Pro to see the approach for this follow-up.
QHow would you handle the case where the agent fails at step 3 but recovers at step 7 and ultimately succeeds?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Scoring only the final task outcome and reporting a single pass/fail number. This tells the PM that 38% fail but not whether the failures are concentrated in one tool, one step position, or one error type, making it impossible to prioritize fixes.