What does observability mean for production LLM agents, and how is it different from app monitoring?
Agent observability adds per-task step counts, tool-call distributions, token cost, trajectory quality, and refusal rate on top of the latency, error rate, and throughput dimensions of traditional app monitoring.
Imagine running a restaurant. App monitoring asks the standard questions: how long did each table wait, how many orders were sent back, how many guests did you serve. For an LLM agent, those questions still matter but they miss most of what can go wrong. Did the waiter take fifteen trips to the kitchen for a single table when two would do? Did the chef ignore half the order and the customer ate it anyway out of politeness? Did the bill come out three times the menu price because the waiter kept asking the chef expensive clarifying questions? Agent observability asks all of those. The dish came out and the table did not complain, but the workflow underneath might still be deeply broken.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Agent observability is the discipline of making a non-deterministic LLM-driven system inspectable: knowing what it did, how well it did, what it cost, and where it broke, without having to redeploy to add new logs. It is the layer above tracing. Tracing tells you everything that happened on one task. Observability turns that raw record into dashboards, alerts, and the ability to ask new questions of the past.
The reason the term needs its own treatment is that classical application monitoring, the kind that watches HTTP endpoints and reports latency, error rate, and throughput, only catches part of what can go wrong with an agent. An agent task can complete with a 200 status, a normal latency, and no exceptions, while producing a wrong answer, looping twenty times, and quietly burning twenty dollars. None of the standard signals see that.
This explanation walks through what observability actually consists of in the agent setting, why the unit of observation shifts from the HTTP request to the agent task, which agent-specific signals matter on top of the classical golden signals, and how teams put it all together in a production stack.
Why traditional APM is not enough
Application performance monitoring tools (Datadog APM, New Relic, Dynatrace, the open-source OpenTelemetry stack) are built around the assumption that a request is a small, fast, stateless thing. The signals they track reflect that: how long a request takes (latency), what fraction of requests fail (error rate), and how many you handle per unit time (throughput). For a microservice fronting a database, these signals capture almost everything that matters.
An agent breaks every assumption in that model. One HTTP request from the user spawns a tree of LLM calls and tool calls under the hood. The request can complete cleanly while the agent inside took twenty steps when it should have taken three, or hit an expensive model when a cheaper one would do, or produced an output the user finds wrong but the system records as success.
Three concrete failure modes show why APM is not enough. The first is the silent wrong answer: no exception, 200 OK, but the answer is incorrect. Latency and error rate report perfection. The second is the runaway loop: the task does eventually return, so latency drifts up rather than spikes, and error rate is unchanged, but cost triples. The third is the refusal cascade: a downstream tool starts erroring and the agent quietly falls back to telling the user 'I cannot help with that' on every request. From the HTTP layer, refusals look like successful responses.
Observability for agents has to surface all three. It does not replace APM; it sits next to it and watches different things.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Langfuse dashboards combine per-task token cost, step count, and LLM-as-judge quality scores into a single agent observability surface.
- LangSmith reports per-chain latency and per-tool error rate alongside model token usage, tied to the LangGraph or LangChain runtime in use.
What an interviewer would ask next. Try answering before peeking at the approach.
QYour task latency is fine but per-task cost has doubled over a month. Which agent observability signal would have caught this earliest?
Step count per task. A doubling of cost with stable latency means the agent is taking more steps, or hitting more expensive tools per task. Step count trends out before cost does because cost depends on which steps and how big their prompts are.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Believing traditional APM is enough because the agent is wrapped in an HTTP endpoint. Latency and error rate hide the failure modes that matter: silent wrong answers, runaway step counts, and quietly-tripled token spend.
60 second bullets to scan on the way to the call.
Define agent observability as tracing plus logging plus metrics for non-deterministic LLM loops.
Name the three traditional golden signals and why they are insufficient for agents.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.