Design a 'max steps' alarm from agent trace data
Your agent occasionally runs to its max_steps cap and returns a bad answer. Design a 'max steps hit' alarm from trace data alone, including what you log, what you alert on, and how you avoid false positives from genuinely hard tasks.
Tag every agent run with steps_taken, max_steps, and outcome; alert on a rolling rate of max_steps_hit, stratified by task type so hard tasks do not drown out real regressions.
Picture a delivery driver with a strict 20-stop limit per shift. If a driver finishes the shift with stops left, the day went well. If they hit the cap before finishing, that is a warning sign. But sometimes the route really has 25 deliveries because of a holiday rush, so you should not panic the first time. You set up a logbook with three columns: stops attempted, the cap that day, and whether the driver finished. Then you watch the running average for each route type. Suburban routes hitting the cap once a week is fine. Downtown routes hitting it three times in a row means something changed and the dispatcher needs to look. The cap itself is not the alarm; the rate at which it is hit, compared to the normal rate for that kind of work, is.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Agent step caps are a safety belt, not an error condition. They exist because a misrouted agent can otherwise loop forever and burn tokens at full throttle. The instrumentation question is not 'how do I alert when the cap fires' but 'how do I tell a real regression from a hard task that legitimately needed all its steps'.
The answer factors into three pieces: a tight span shape that records enough to compute any downstream slice, a rate-based alarm that knows about base rates, and a stratification that defeats Simpson's paradox so a normal hard-task rate does not mask a true regression elsewhere.
This card walks through each piece, then names the operational hooks (drill-down links, leading indicators) that make the alarm actually actionable rather than another notification the on-call mutes.
Span shape: three attributes that carry the whole alarm
On the root agent span, record three attributes. agent.steps_taken is the integer count of agent loop iterations that actually executed. agent.max_steps is the cap value that was configured for that run; it has to be on the span because the cap can change between deploys and you need to compare against the cap of record. agent.outcome is an enum with at least three values: succeeded, max_steps_hit, and errored.
Why outcome is not just a boolean
A two-value field collapses two failure modes that need different responses. errored covers tool failures, parsing failures, and provider 5xx; the on-call response is investigate and fix. max_steps_hit is graceful degradation; the response is rate analysis then tune. Pooling them hides which one is rising.
What to add for richer slicing
Tag agent.task_type (or run a classifier at entry), agent.prompt_version, and agent.tool_surface (the set of tools available for that run). Each child step span carries the tool name and step index, which lets you diagnose tool-thrashing once you have drilled into a specific failing trace.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Langfuse trace UI exposes custom span attributes as filterable columns, which lets a team define this exact stratified view without code changes.
- LangSmith's agent-trace view renders steps as a tree and surfaces a 'max-iterations reached' flag teams routinely turn into a dashboard tile.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat if you do not know task_type at entry, only after the agent has run?
Either run a cheap classifier on the input prompt before the agent starts, or tag task_type retroactively from the agent's first plan step and join in the analytics query.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating every max-steps hit as a P1 incident; the cap is a graceful degradation control, not an error, and false positives drown the on-call in noise.
60 second bullets to scan on the way to the call.
Attributes to record on the root agent span
Why max_steps_hit is a graceful outcome not an error
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.