Pick the right Phoenix workflow when you want LLM-as-judge results back as a column on every trace

MCQ·Medium·4.0 · 0·~1 min·Asked atDoordash Promptlayer Turing

Attempt it

**Phoenix uses an offline on traces pattern** for LLM-as-judge evaluation. It does not intercept your live requests; instead, it pulls already-recorded spans into a Python DataFrame, runs your judge against each row, and writes the resulting labels and scores back onto the original spans. The three-step shape: 1. **Pull spans into a DataFrame.** Phoenix provides query helpers (`get_spans_dataframe` and similar) that filter by time range, project, span kind, attributes. You end up with a pandas DataFrame where each row is a span and columns hold the prompt, completion, retrieved context, etc. 2. **Run the evaluator.** Phoenix ships `llm_classify` for classification-style judges and `run_evals` for the higher-level orchestration. Prebuilt evaluators cover common rubrics: hallucination, relevance, toxicity, retrieval-precision. The evaluator adds score and label columns to the DataFrame. 3. **Call `log_evaluations`.** This writes the scores back to Phoenix as annotations on the original spans. Once written, the trace UI shows them as filterable, sortable, chartable columns alongside latency and cost. The answer is option A. Option B (in-line via a header) is wrong because Phoenix does not run the judge synchronously on every request; that pattern would double latency and double cost for every call. Option C (re-train the ingester) is nonsense; the ingester does not need retraining. Option D (human-only) is wrong; LLM-as-judge is a first-class workflow.

**Phoenix runs LLM-as-judge offline on already-captured traces** and writes the resulting scores back as span annotations. The three-step shape: 1. **Pull spans into a DataFrame.** `get_spans_dataframe` (and friends) returns a pandas DataFrame filtered by project, time window, and span kind. Each row has the prompt, completion, attributes, and any retrieved context. 2. **Run the evaluator.** `llm_classify` runs a classification judge (your prompt template plus a model like GPT-5.5 or Claude Opus 4.7). Phoenix also ships prebuilt evaluators for common dimensions: `HallucinationEvaluator`, `RelevanceEvaluator`, `QAEvaluator`. The evaluator returns a DataFrame with score and label columns matching the row count. 3. **`log_evaluations(SpanEvaluations(eval_name='hallucination', dataframe=results_df))`.** This writes the scores back to Phoenix, attached to the original span ids. From that point on, the trace UI surfaces the eval as a column; you can filter ('show me all traces where hallucination = yes'), sort, and chart distributions. **Why offline on traces rather than in-line:** - **Latency**: in-line would double the wall-clock of every user request. The judge is a full LLM call. - **Cost**: judging every request is often 30 to 100 percent of the cost of the request being judged; sampling makes the math work. - **Iteration**: judges themselves need calibration. Offline lets you re-judge the same trace dataset with v2 of your rubric without affecting production. The pattern is the same in Langfuse 3.x (evaluations attach as scores), in LangSmith (annotations), and in Braintrust (eval logs). Phoenix's specific API names are `llm_classify`/`run_evals` and `log_evaluations`; the conceptual shape is identical.

TL;DR

Pull spans into a DataFrame, run llm_classify or a prebuilt evaluator, then call log_evaluations to annotate the original spans. Offline-on-traces, not in-line.

Memory aid

Easy to grasp

Imagine you have a stack of cooking videos. You want to label each one as 'good recipe' or 'too greasy.' You first download the videos into a folder, watch them and write the label in a spreadsheet next to each video name, then upload the labels back to the video service so when you browse, every video shows its label. Phoenix works the same way. It pulls your LLM traces into a table, an evaluator labels each row, then those labels get written back to the original traces so they show up as a filterable column in the trace browser. The judging happens after the fact, on a snapshot of traces, not on every request as it runs.

Key concepts

Observability & Tracing

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Arize Phoenix is the open-source LLM-observability product in the Arize ecosystem, popular for RAG evaluation and embedding-drift visualization. Its evaluation model is deliberately offline on traces: production captures spans, an evaluator runs against a snapshot, and the resulting scores are written back as span annotations that surface in the trace UI. The pattern is initially surprising for teams expecting in-line judging, but it is the right shape for cost, latency, and judge-iteration reasons.

This walkthrough explains the three-step API contract (pull, llm_classify, log_evaluations), the reasons offline judging beats in-line for production quality measurement, the prebuilt evaluators Phoenix ships, the operational pattern for keeping judge runs sampled and calibrated, and the workloads where Phoenix is the wrong tool.

Mental model: Phoenix evaluates traces after they happen, on a snapshot DataFrame, and writes scores back as queryable annotations. It is not a request interceptor. It is a batch quality-measurement layer on top of trace capture.

The three-step API: pull, classify, log

Step 1: pull spans into a DataFrame

Phoenix exposes get_spans_dataframe and family. The call takes filters (project, time range, span kind, attribute predicates) and returns a pandas DataFrame where each row is a span. Columns include:

span_id (the join key).
name, start_time, end_time, status_code.
attributes (often expanded into individual columns like input.value, output.value, llm.model_name).
RAG-specific columns: retrieval.documents and retrieval.scores.

The DataFrame is the working surface for the next step.

Step 2: run an evaluator

Two APIs cover most use cases:

llm_classify(dataframe, template, model, rails): a generic classification judge. You supply the prompt template, the judge model (often a flagship like Claude Opus 4.7 or GPT-5.5), and the set of allowed labels (rails). Returns a DataFrame with label and score columns plus optional explanation.
run_evals(dataframe, evaluators=[...]): orchestrates multiple evaluators (often prebuilt ones). Returns a DataFrame per evaluator.

Prebuilt evaluators cover common rubrics:

HallucinationEvaluator: detects ungrounded statements in RAG outputs.
RelevanceEvaluator: scores whether retrieved chunks are relevant.
QAEvaluator: scores whether the answer addresses the question.
ToxicityEvaluator: flags harmful content.

Custom rubrics use llm_classify with your own template.

Step 3: log_evaluations writes back

python

from phoenix.trace import SpanEvaluations
px.log_evaluations(
    SpanEvaluations(eval_name='hallucination', dataframe=results_df),
)

This writes the evaluations as annotations on the original spans (joined by span_id). From now on, the trace UI surfaces the eval as a column. You can filter ('show traces with hallucination=yes'), sort, and chart distributions.

Forgetting this step is the most common bug. Your DataFrame has the scores but the UI never sees them. Always log_evaluations.

Why offline on traces beats in-line

Production patterns and judge calibration

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Arize Phoenix 5.x's documentation gives the exact pattern with llm_classify and log_evaluations.
Teams running RAG with self-hosted Llama 4 Maverick or DeepSeek V4 commonly use Phoenix for offline evals on production traces.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you scale this from 1 percent sampling to 100 percent for a small set of critical traces?

Add a high-priority filter in the DataFrame query (e.g. spans where user_tier='enterprise' OR error=true). Run the judge on all of those plus the sample of the rest. Stratified sampling at the DataFrame layer is the natural place.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Forgetting log_evaluations. You get a DataFrame with scores but the trace UI never sees them, so dashboards stay blank.

60 second bullets to scan on the way to the call.

The three steps: pull, classify, log_evaluations
Why offline on traces beats in-line judging

Primary sources. Browse if you want the original framing.