What does observability mean for production LLM agents, and how is it different from app monitoring?

Observability is the ability to ask any question about what your system did, after the fact, without going back to add new logs. For a regular web service, you usually need three numbers: how fast each request was, how many failed, and how many you served per second. An LLM **agent** needs more. The request might return a 200 with normal latency, yet the agent's answer was wrong, or it spent twenty dollars on a single task, or it looped twelve times when one pass should have been enough. None of that is visible from latency or error rate. So agent observability adds new dimensions on top of the old ones. Step count per task tells you when the agent is wandering. Tool-call distribution tells you which tools the agent depends on and which it never uses. Token cost per task tells you when spend creeps up. Trajectory quality tells you whether the answer was good. Refusal or fallback rate tells you when the agent is bailing out. The other shift is the unit of measurement. App monitoring counts HTTP requests. Agent observability counts agent tasks, because one task can contain dozens of model and tool calls that each have their own latency and cost.

**Agent observability** is tracing plus structured logging plus metrics, but framed around the agent task as the unit of observation rather than the HTTP request. Traditional app monitoring measures three headline things: request latency, error rate, and throughput. These still apply, but they capture only the surface of an agent system. A request can return 200 with normal latency while the agent silently produces a wrong answer, loops twenty times, and burns twenty dollars in tokens. The extra dimensions an agent dashboard needs are specific. Step count per task surfaces wandering loops. Tool-call distribution shows which tools the agent leans on and which it ignores. Token and dollar cost per task surface runaway spend. Trajectory quality (often a scored sample of completed tasks) tracks whether the agent actually solves the problem. Refusal and fallback rates surface when the agent is bailing out instead of acting. The unit shift matters. App monitoring sums over requests; agent observability sums over tasks, because one task fans out into many requests under the hood. Production teams pair this with per-task tracing so any aggregate metric drills down into a concrete trace. Tools like Langfuse, LangSmith, Arize Phoenix, Helicone, and Weave wrap this whole bundle.