List the four observability signals every mature LLMOps stack relies on

Flashcard·Easy·4.0 · 0·~30s·Asked atCred Hcl IBM

Attempt it

Traces and spans following the OpenTelemetry GenAI conventions (gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.response.finish_reasons).

Online sampled evaluations on a small slice of production traffic, with judge scores written back as span attributes.

Golden-set regression gates run in CI on every prompt or model change, with a paired comparison and significance test.

Drift alerts on rolling baselines of judge scores, refusal rates, and per-tenant breakdowns so cohort regressions surface before they aggregate-mask.

The observability topic covers the instrumentation layer in depth; LLMOps consumes those four signals as the data backbone for canary read-out, cost dashboards, incident response, and post-incident eval expansion.

Your answer

**LLMOps and LLM observability are two views of the same plumbing.** Observability is the data layer; LLMOps is the discipline that uses the data to ship and operate. Mature stacks rely on four signals. **Traces and spans** following OpenTelemetry GenAI conventions. Every LLM call becomes a span with attributes like model, input tokens, output tokens, finish reason, and provider. Multi-step agents produce a tree of spans showing the call graph. Traces are the per-request truth. **Online sampled evaluations.** A small slice of production traffic (typically 1 to 10 percent) is scored live by a judge model (LLM-as-judge) or a heuristic evaluator. The score is written back as a span attribute so you can dashboard it. **Golden-set regression gates in CI.** Every prompt PR replays a fixed evaluation set, compares scores against baseline with a significance test, and blocks merge on regression. This is the pre-deploy gate. **Drift alerts on rolling baselines.** Continuous monitoring of judge scores, refusal rates, and per-tenant breakdowns to catch cohort regressions that the aggregate average hides. All four feed into the same dashboards: canary read-out, cost per request, incident response, and post-incident dataset expansion. Without any one signal, a class of failure ships silently.

**The four signals are traces, online evals, CI regression gates, and drift alerts.** Each catches a different failure mode; the discipline is that all four are running at once. **Traces and spans (per-request truth).** OpenTelemetry GenAI semantic conventions standardize the schema: `gen_ai.system`, `gen_ai.request.model`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.response.finish_reasons`. Auto-instrumentors from Langfuse, LangSmith, Phoenix, and Datadog emit these automatically for major SDKs. Multi-step agents produce a span tree that shows the call graph. Traces are how you debug a single failure. **Online sampled evaluations (continuous quality signal).** A 1 to 10 percent sample of production traffic is scored by an LLM-as-judge or a custom evaluator. The scores are written back as span attributes (`eval.faithfulness`, `eval.judge_score`). Dashboards aggregate them by feature, by tenant, by model. This is how you spot a regression in flight. **Golden-set CI regression gates (pre-deploy gate).** Every prompt or model PR runs against a fixed golden set, compares scores to baseline with a paired significance test, and posts a delta table plus worst-N failing examples to the PR. Merge is blocked on regression. This is how you stop bad changes shipping. **Drift alerts on rolling baselines (cohort and trend safety net).** Continuous monitoring of judge scores, refusal rates, and per-tenant breakdowns against rolling baselines. A 7-day judge-score median that drifts down by 5 percent triggers an investigation. Cohort regressions (one tenant, one document type) surface before they aggregate-mask into the global average. **Why all four matter.** Traces alone tell you what happened on one call but not whether quality is good. Online evals alone catch in-flight regressions but cannot prevent bad changes shipping. CI alone catches PR-time regressions but misses model drift and provider behavior shifts.

TL;DR

Mature LLMOps stacks rely on four signals: OTel-style traces, online sampled evals, golden-set CI regression gates, and drift alerts on rolling baselines.

Memory aid

Easy to grasp

Picture running a busy restaurant kitchen. You need four pieces of feedback. First, a live order ticker so you see every dish in flight (traces). Second, a head chef randomly tasting plates as they go out (online evals). Third, a tasting panel that runs new recipes against a fixed menu before they ship (CI golden set). Fourth, a manager watching weekly trends to spot when one chef's dishes are slowly getting saltier (drift alerts). Without any one of the four, you find out the kitchen has problems from a bad review instead of from your own dashboard.

Key concepts

LLMOps

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Mature LLMOps stacks rely on four data signals that together cover the failure modes of an LLM-powered service: traces, online sampled evaluations, golden-set regression gates in CI, and drift alerts on rolling baselines. Two are continuous (traces, online evals) and two are triggered (CI gate runs on PRs, drift alerts fire on threshold breach). All four feed into the same dashboards and the same closed loop, where production failures grow the eval set and the eval set hardens the gate.

LLM observability and LLMOps are not two separate disciplines; they are the same plumbing seen from two angles. Observability is the data layer that emits the signals; LLMOps is the operational discipline that consumes them to gate deploys, run incidents, and ship improvements. This walkthrough breaks down each signal, what it catches, what it misses, and how the four wire together into a coherent operational loop.

Mental model: four signals, four blind spots if you drop one. Traces give you per-call truth; online evals give you in-flight quality; CI gates give you pre-deploy safety; drift alerts give you trend and cohort safety. Together they close the loop.

Traces and spans: the per-request truth

OpenTelemetry GenAI conventions

The OpenTelemetry GenAI semantic conventions standardize the trace attribute schema across providers and tools. The canonical attributes:

gen_ai.system (provider: openai, anthropic, google, etc.)
gen_ai.request.model (model id at the API)
gen_ai.request.temperature, gen_ai.request.top_p, gen_ai.request.max_tokens
gen_ai.usage.input_tokens, gen_ai.usage.output_tokens
gen_ai.response.finish_reasons (stop, length, content_filter, tool_calls)
gen_ai.response.id (provider request id)

Auto-instrumentors from Langfuse, LangSmith, Phoenix, Datadog LLM Observability, and Honeycomb emit these attributes for the major SDKs (OpenAI Python, Anthropic Python, Google Generative AI, the Vercel AI SDK). The cross-vendor schema means a trace from one tool can be ingested by another.

Multi-step span trees

Agents and chains produce span trees: an outer chain span with child spans for each LLM call, tool call, retriever call. The tree structure is essential for debugging multi-step failures because the parent span carries the user-facing context while the child spans carry the per-step detail.

Sampling strategy

Full-trace ingestion at production scale gets expensive fast. Two patterns:

Head sampling. Decide at ingest whether to keep the trace. Saves cost but loses error traces randomly.
Tail sampling. Buffer at the collector, decide after seeing the full trace. Keeps all error traces, downsamples successful traces.

Most production setups use tail sampling for errors plus head sampling for the success path.

Online sampled evaluations: the continuous quality signal

CI regression gates and drift alerts: the two trigger-based signals

The closed loop and the failure-mode map

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

OpenTelemetry GenAI semantic conventions are the cross-vendor schema for trace attributes, adopted by Langfuse, LangSmith, Phoenix, Datadog, and Honeycomb.
Anthropic and OpenAI both document online sampled eval patterns where a judge model scores a percentage of production traffic and writes scores back as metadata.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you decide the sample rate for online evals on a high-traffic feature?

Start with the eval budget (judge cost you can absorb). Divide by judge cost per call to get max samples per day. Convert to a sample rate against current QPS. Floor at 1 percent for a usable signal; ceiling where judge cost overtakes generation cost. Bias sampling toward flagged cohorts and known failure prone routes.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating traces alone as observability. A trace tells you what happened on one call; you need the other three signals to know whether the system is healthy in aggregate.

60 second bullets to scan on the way to the call.

Which four signals does a mature LLMOps stack carry, and what does each catch?
Which OpenTelemetry GenAI attributes belong on every LLM span?

Primary sources. Browse if you want the original framing.