Design a diagnostic eval for an agentic workflow with 62% task success across 5 tools and 10+ steps that reveals WHY tasks fail, not just whether they pass.

Short answer·Hard·4.0 · 0·~3 min·Asked atHaptik Hebbia Snorkel Ai·Relevant atAnyscale Databricks Datadog Scale Ai

Attempt it

Free · 2 AI evals / day

A 62% success rate on a multi-step agent workflow does not explain why 38% of tasks fail. The eval needs to go deeper. **Step-level scoring** checks each step independently. Was the right tool called? Did the arguments match the schema? Was the result interpreted correctly? Deterministic checks catch mechanical errors (wrong tool name, missing arguments). LLM-as-judge catches reasoning errors (wrong tool for the sub-task, bad argument values). **Trajectory efficiency** compares the agent's step count to an oracle (minimum-step solution). Extra steps reveal planning failures, loops, and redundant calls. **Error attribution** traces backward from a failed outcome to the first wrong step. Each root cause goes into a failure taxonomy: wrong tool selection (roughly 33%), correct tool with wrong arguments (25%), misinterpreted result (20%), hallucinated tool (12%), infrastructure timeout (10%). The taxonomy tells engineering where to invest: if one-third of failures are wrong tool selection, improve tool descriptions and add few-shot examples. Two diagnostic views complete the picture: per-tool success rate (which tools are problematic?) and per step position failure rate (where in the trajectory do errors concentrate?).

**Three scoring layers.** (1) Step-level correctness: deterministic checks (tool name exists, arguments match schema) plus LLM-as-judge semantic checks (right tool for the sub-task, reasonable argument values). (2) Trajectory efficiency: compare agent step count to oracle trajectory, flag redundant steps, loops, and unnecessary calls. (3) Error attribution: trace backward from failed outcome to the first wrong step. **Failure taxonomy.** Wrong tool selection (roughly 33%): fix with better tool descriptions and few-shot examples. Correct tool, wrong arguments (25%): fix with better documentation and schema validation. Misinterpreted result (20%): fix with structured output parsing. Hallucinated nonexistent tool (12%): fix with tool-name validation and prominent tool-set listing. Infrastructure timeout (10%): fix with retry logic and fallback paths. **Diagnostic views.** Per-tool success rate reveals which tools the agent struggles with. Per-step position failure rate reveals where in the trajectory errors concentrate. If step 3 has a 25% failure rate while other positions are below 10%, the agent has a consistent planning breakdown at that trajectory depth. **Existing patterns.** SWE-bench Verified, TAU-bench, and AgentBench already use trajectory scoring. Adapt their rubric patterns to the specific tool set. The taxonomy drives engineering priorities directly: percentages map to which fixes yield the highest return.

TL;DR

Three scoring layers (step-level correctness, trajectory efficiency, backward error attribution) plus a failure taxonomy that tells engineering exactly where to invest to move the 62% success rate.

Memory aid

Easy to grasp

Imagine a cooking competition where contestants follow a 10-step recipe using 5 different kitchen tools. If 38% of dishes taste bad, the judges need to know why. Did the chef pick the wrong tool (used a blender instead of a mixer)? Did they use the right tool with wrong settings (oven at 500 degrees instead of 350)? Did they misread the timer? By watching each step and noting where things first went wrong, you can tell the chef exactly what to practice instead of just saying the dish failed.

Key concepts

Agent Evaluation SWE-bench

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Evaluating agentic workflows is fundamentally different from evaluating single-turn LLM outputs. A single-turn eval scores one output against one reference. An agent eval must score a trajectory: a sequence of decisions, tool calls, and result interpretations that unfolds over 10 or more steps.

The pass/fail success rate (62% in this scenario) is a business metric. It tells the PM how often the agent succeeds. It does not tell anyone why the agent fails. The 38% failure rate is a single number hiding a distribution of different failure types, concentrated at different steps, involving different tools. Without decomposing that 38%, the team cannot prioritize which fix to build next.

The design in this answer layers three scoring mechanisms (step-level correctness, trajectory efficiency, backward error attribution), builds a failure taxonomy that maps directly to engineering actions, and defines two diagnostic views (per-tool success rate, per step position failure rate) that surface patterns invisible in the aggregate.

Step-level correctness: deterministic plus semantic scoring

Step-level scoring evaluates each step in the trajectory independently, asking three questions. Did the agent call the right tool? Were the arguments valid? Was the result interpreted correctly?

Deterministic checks handle the mechanical parts. Tool-name validation checks whether the called tool exists in the available set. If the agent calls search_database_v2 but the tool set only contains search_database, the call fails before execution. Argument schema validation checks whether required fields are present, types are correct, and values are within acceptable ranges. These checks run instantly, catch mechanical errors (hallucinated tools, missing arguments, type mismatches), and produce unambiguous pass/fail signals.

Semantic scoring with LLM-as-judge handles the reasoning parts. Was this the right tool for the current sub-task? The agent might call a file-search tool when a database-query tool would be correct. Both tools exist; the choice is wrong. Were the argument values reasonable given the context? The agent might call the right tool with a query string that does not match the user's intent. These are judgment calls that deterministic checks cannot make.

The combination is important. Deterministic checks alone miss reasoning errors. Semantic checks alone miss mechanical errors (an LLM-as-judge might not flag a hallucinated tool name if the name sounds plausible). Running both catches the full error surface.

The per-step scores feed into the trajectory-level analysis. A trajectory where step 3 is wrong and steps 4 through 10 are downstream consequences looks different from a trajectory where steps 3, 7, and 9 are independently wrong. The step-level scores enable that distinction.

python

# Step-level deterministic validation (pseudocode)
def validate_step(step, tool_registry):
    # Check tool exists
    if step.tool_name not in tool_registry:
        return StepResult(
            valid=False,
            error_type="hallucinated_tool",
            detail=f"{step.tool_name} not in registry"
        )
    
    tool = tool_registry[step.tool_name]
    
    # Check argument schema
    schema_errors = tool.schema.validate(step.arguments)
    if schema_errors:
        return StepResult(
            valid=False,
            error_type="schema_violation",
            detail=schema_errors
        )
    
    return StepResult(valid=True, error_type=None)

Trajectory efficiency: oracle comparison and redundancy detection

Backward error attribution and the failure taxonomy

Diagnostic views: per-tool and per step position analysis

Adapting existing benchmark patterns to production agent eval

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

SWE-bench Verified evaluates software-engineering agent trajectories with per-step analysis of code edits, using test suite execution as the deterministic correctness check and trajectory length as an efficiency metric.
TAU-bench scores customer-service agent dialog per turn, checking rule-following and tool-call validity at each step, directly implementing the step-level correctness layer described here.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you build the oracle trajectory for tasks where the optimal solution is not obvious?

Three approaches, each with tradeoffs. Human annotation: have domain experts solve a sample of tasks and record the trajectory. Strongest model: use the most capable model available with a verbose planning prompt and validate the trajectory. Retrospective optimization: for tasks the agent solved successfully, find the shortest successful trajectory across multiple runs and use it as the oracle. Human annotation is the gold standard but expensive. Strongest-model oracles are scalable but can be wrong. Retrospective oracles only work for tasks the agent can already solve.

3 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Scoring only the final task outcome and reporting a single pass/fail number. This tells the PM that 38% fail but not whether the failures are concentrated in one tool, one step position, or one error type, making it impossible to prioritize fixes.

60 second bullets to scan on the way to the call.

Name the three scoring layers: step-level correctness, trajectory efficiency, backward error attribution
Explain step-level scoring with both deterministic (schema validation) and semantic (LLM-as-judge) checks

Primary sources. Browse if you want the original framing.