Why is final-task success rate insufficient for evaluating LLM agents, and what does trajectory eval add?

An **LLM agent** solves a task by taking many steps: it plans, calls tools, reads results, and acts again until it finishes. The simplest way to score it is final-task success rate: did the whole task get done, yes or no? That single number is weak for two reasons. First, it is sparse. Hard agent tasks fail most of the time, so a small set of runs gives you a noisy percentage that bounces around. Second, it is silent about why. If the agent failed a ten-step task, the success number does not tell you whether step two or step seven broke. Trajectory evaluation fixes this by scoring the intermediate steps, not just the end. It records every action the agent took and grades each one. Four metrics show up again and again. Tool call accuracy: did the agent pick the right tool with the right arguments? State validity: after each action, is the world in the expected state? Efficiency: how many redundant or wasted actions did it take? Error recovery: after a failed call, did the agent adapt or spiral? With these, debugging becomes targeted. If tool call accuracy drops at step three across many runs, you know exactly where to look.

Final-task success rate is a binary, end of episode signal: did the agent finish the task or not. It fails as a primary metric for two reasons. It is sparse and high variance. Hard agent benchmarks fail most of the time, so a handful of runs gives a noisy percentage. Worse, it offers no failure localization. A failed ten-step task tells you nothing about whether step two or step seven broke. Two agents at the same success rate can differ wildly. One may use five clean tool calls; another fifty redundant ones. **Trajectory evaluation** scores the intermediate steps. It treats the run as a sequence of actions, not one outcome. The standard step-level metrics start with tool call accuracy (right tool, right parameters) and state validity (world state matches the expected state after each action). Then come efficiency (count of redundant or no-op actions) and error recovery (does the agent adapt after a failed call). These unlock targeted debugging. A drop in tool call accuracy at step three across many trajectories points to a specific planning or prompting failure that a global pass-fail number would mask. You can also award partial credit on a ground-truth trajectory, turning a sparse binary into a graded signal that discriminates between near-misses and total failures.