An LLM call isn't like a normal API call. The same input can produce different outputs, the quality depends heavily on the exact prompt, and each call costs money and takes a variable amount of time. So your trace has to capture enough to reproduce and explain any single call after the fact.
That means three groups of fields. First, the **full prompt** you actually sent — system prompt plus all the context — and the model's complete output. Second, the model name and version, the input and output token counts, and the cost you computed from them. Third, the latency (time to first token and total) and a request or trace ID so you can correlate this call with logs, user reports, and downstream effects.
The two wrong options drop the prompt to save space. Since outputs are prompt-dependent and non-deterministic, throwing the prompt away means you can never reproduce a bad answer.
The job of an LLM trace is to let you fully reconstruct and diagnose any call long after it happened. Because outputs are stochastic and prompt-dependent, and because cost and latency vary per call, you log a specific bundle.
The **prompt and output** are non-negotiable: the exact assembled prompt — system instructions, retrieved context, user message — and the model's complete response. Without the input you cannot reproduce the output; with a paraphrased or summarized prompt you reproduce something else. Next, model name and version, because provider model updates silently shift behavior, and a trace that doesn't pin the version can't be re-run. Then token counts and computed cost, which drive billing, budgets, and routing decisions. Finally, latency split into time to first token and total, plus a correlation ID that threads this call through your other logs and any user report.
The distractors both omit the prompt — one to "keep storage small," one by calling prompts an implementation detail. Both are fatal. Storage is cheap relative to the cost of an unreproducible production incident, and the prompt is the single most diagnostic field you have. Capturing only the final answer is logging the symptom and discarding the cause.
Quick reference
TL;DR
Trace every LLM call with the full prompt and output, model name/version, token counts and cost, latency, and a correlation ID — or you can't reproduce production bugs.
Imagine a chef who cooks a slightly different dish every time, even from the same order. A customer complains a meal was awful. If all you wrote down was 'served a meal,' you're stuck — you can't recreate it. So you keep a full ticket: the exact recipe used, which chef, how long it took, how much it cost, and an order number. Now you can pull the ticket and see exactly what went wrong. An LLM trace is that ticket: the prompt sent, the answer, the model, the tokens, the time, and an ID to find it all.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Prompts containing regulated PII or secrets under strict retention rules You can't store raw prompts indefinitely; redact, tokenize, or hash sensitive spans and shorten retention rather than logging everything verbatim.
Extreme-throughput, low-value calls where full-fidelity tracing is cost-prohibitive Storing every full prompt/output at millions of calls/day can exceed the workload's value; sample full traces and keep lightweight metadata on the rest.
Hard real-time paths where synchronous trace writes add unacceptable latency Inline blocking writes can hurt p99; emit traces asynchronously off the critical path instead of skipping fields.
Quick scan · 2–4 min
Everything important, quickly.
Spend 5-6 minutes: anchor on the 'can I re-run this exact call?' test, then justify each field group and explain why dropping the prompt is fatal.
Side-by-side
Field group
Why it's logged
Cost of omitting
Full prompt + output
Exact input/output for replay
Can't reproduce a bad answer
Model name + version
Pin behavior across provider updates
Silent drift is unattributable
Token counts + cost
Budgets, caps, routing
No cost visibility or control
Latency (TTFT + total)
Perf and streaming UX
Regressions go unnoticed
Trace / correlation ID
Thread across services and reports
Logs can't be joined
Real-world examples
Real products, models, and research that use this idea.
1LangSmith capturing full prompt/response, model, token usage, cost, and latency per call with a trace ID for replay.
2Langfuse tracing nested LLM and tool spans across an agent run, correlating them under one session ID.
3OpenLLMetry / OpenTelemetry GenAI semantic conventions standardizing model, token, and latency attributes on spans.
4Provider dashboards (OpenAI, Anthropic usage logs) exposing token counts and per-request cost for budget tracking.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you trace prompts that contain user PII without violating retention or privacy rules?
Sign in or go Pro to see the approach for this follow-up.
QAt very high QPS, full traces get expensive — how do you decide what to sample?
Sign in or go Pro to see the approach for this follow-up.
QHow does the trace connect to your eval and feedback loops?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Logging only the final user-visible answer to save storage. Without the exact prompt and model version, you can't reproduce or explain a bad response.