Trace every LLM call with the full prompt and output, model name/version, token counts and cost, latency, and a correlation ID — or you can't reproduce production bugs.
Imagine a chef who cooks a slightly different dish every time, even from the same order. A customer complains a meal was awful. If all you wrote down was 'served a meal,' you're stuck — you can't recreate it. So you keep a full ticket: the exact recipe used, which chef, how long it took, how much it cost, and an order number. Now you can pull the ticket and see exactly what went wrong. An LLM trace is that ticket: the prompt sent, the answer, the model, the tokens, the time, and an ID to find it all.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Observability is the unglamorous backbone of every LLM product that survives contact with real users. It rarely shows up in architecture diagrams, but the first time someone reports "the assistant said something completely wrong yesterday," the quality of your tracing decides whether you can investigate or just shrug. This question probes whether you know what a usable trace actually contains.
The reason the answer isn't obvious is that LLM calls violate the assumptions behind normal request logging. A normal endpoint is deterministic: given the request, you can re-derive the response, so logging the response is often enough. An LLM call is stochastic and prompt-dependent — the same code can produce different outputs, and tiny prompt differences swing quality hard. The response alone tells you nothing about why.
This deep dive builds the trace from one test — can I reproduce this exact call later? — derives each field group from it, and then shows why the two distractors, both of which drop the prompt to save space, are the specific mistakes that leave teams blind during their first production incident.
The one test that generates the whole field list
Anchor everything on a single question: if a user reports a bad answer tomorrow, can I pull one record and re-run the exact call that produced it? Every field in a good trace exists to make that answer yes.
To re-run a call you need the exact input, so you log the full assembled prompt — system instructions, retrieved context, conversation history, the user turn — as the model actually received it. Not a template, not a summary; the bytes. You also need the model identity, because the same prompt against a different model or a silently updated version gives a different output, so you pin model name and version.
That reproduces the call. The rest of the fields let you reason about it. Token counts and cost answer "was this expensive, and why?" Latency answers "was this slow, and where?" A correlation ID answers "what else happened around this call?" by threading it to other services, the user session, and any feedback. Notice the method: you don't memorize a checklist, you derive it from the reproduce and explain test. That's also how you'd defend any field an interviewer questions — point back to which half of the test it serves.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Field group | Why it's logged | Cost of omitting |
|---|---|---|
| Full prompt + output | Exact input/output for replay | Can't reproduce a bad answer |
| Model name + version | Pin behavior across provider updates | Silent drift is unattributable |
| Token counts + cost | Budgets, caps, routing | No cost visibility or control |
| Latency (TTFT + total) | Perf and streaming UX | Regressions go unnoticed |
| Trace / correlation ID | Thread across services and reports | Logs can't be joined |
Real products, models, and research that use this idea.
- LangSmith capturing full prompt/response, model, token usage, cost, and latency per call with a trace ID for replay.
- Langfuse tracing nested LLM and tool spans across an agent run, correlating them under one session ID.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you trace prompts that contain user PII without violating retention or privacy rules?
Discuss field-level redaction, tokenization or hashing of sensitive spans, shorter retention tiers, and access controls on the trace store.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Logging only the final user-visible answer to save storage. Without the exact prompt and model version, you can't reproduce or explain a bad response.
60 second bullets to scan on the way to the call.
Why the exact assembled prompt must be logged, not a summary
Why pinning the model version matters under silent provider updates
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.