What does observability mean for production LLM agents, and how is it different from app monitoring?

Attempt it

Your answer

Quick reference

TL;DR

Agent observability adds per-task step counts, tool-call distributions, token cost, trajectory quality, and refusal rate on top of the latency, error rate, and throughput dimensions of traditional app monitoring.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine running a restaurant. App monitoring asks the standard questions: how long did each table wait, how many orders were sent back, how many guests did you serve. For an LLM agent, those questions still matter but they miss most of what can go wrong. Did the waiter take fifteen trips to the kitchen for a single table when two would do? Did the chef ignore half the order and the customer ate it anyway out of politeness? Did the bill come out three times the menu price because the waiter kept asking the chef expensive clarifying questions? Agent observability asks all of those. The dish came out and the table did not complain, but the workflow underneath might still be deeply broken.

Key concepts

Observability Monitoring Tracing

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

Define observability as tracing plus logging plus metrics rooted at the agent task. Contrast the three golden signals of APM with the five agent-specific signals. Explain why one task fans out into many requests and why the unit of observation must move. Close on how aggregate metrics and concrete traces are paired in production tools.

Real-world examples

Real products, models, and research that use this idea.

Langfuse dashboards combine per-task token cost, step count, and LLM-as-judge quality scores into a single agent observability surface.
LangSmith reports per-chain latency and per-tool error rate alongside model token usage, tied to the LangGraph or LangChain runtime in use.
Arize Phoenix surfaces trajectory-level quality scores by joining traces with evaluation runs in the same UI.
Helicone tracks per-user and per-key token and dollar cost on top of every OpenAI or Anthropic call routed through its proxy.
Weave from W&B ties tracing to experiment runs, so a quality regression on a new agent version shows up next to its trace and its eval scores.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QYour task latency is fine but per-task cost has doubled over a month. Which agent observability signal would have caught this earliest?

Sign in or go Pro to see the approach for this follow-up.

QRefusal rate jumps from 1% to 12% overnight. What kind of root cause does that point to, and what signal do you cross-reference?

Sign in or go Pro to see the approach for this follow-up.

QWhy do tracing and observability sit in the same product, even though they are different concepts?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Believing traditional APM is enough because the agent is wrapped in an HTTP endpoint. Latency and error rate hide the failure modes that matter: silent wrong answers, runaway step counts, and quietly-tripled token spend.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.