Walk through how Phoenix surfaces embedding drift for a RAG product

Short answer·Hard·4.0 · 0·~3 min·Asked atIBM Perplexity Persistent

Attempt it

Your RAG product is six months in. Phoenix is logging every query embedding. How does Phoenix help you spot that user queries have drifted away from the distribution your index was built on, and what concrete UI affordance shows it?

Free · 2 AI evals / day

Phoenix is open-source LLM observability with a strong RAG-evaluation toolkit. Embedding drift is one of its flagship features. The core primitive is comparing two collections of embeddings: a **reference** collection (your indexed documents, or an earlier production window of queries) and a **target** collection (recent production queries). Phoenix knows both because you log query embeddings on every retrieval span and you can import your document index. The **UI affordance** is the Embedding tab. It shows a UMAP projection of both collections in a single 2D scatter, color-coded by source (reference vs target). Drift shows up visually as the target color pulling into regions the reference color does not cover, or new sub-clusters appearing in unfamiliar territory. **Quantitatively**, Phoenix reports drift metrics: centroid distance, PSI (Population Stability Index), and Wasserstein distance on per-dimension distributions. A sharp PSI increase is the canonical signal that the distribution shifted. The **remediation** is the actually-useful part. Cluster the drifted region, sample queries, label them, decide whether to refresh the index, retrain the embedder, or add a fallback policy for the underserved cluster.

Phoenix surfaces embedding drift via three things working together: the Embedding tab in the UI, the reference vs target comparison primitive, and a small set of drift metrics. **Reference vs target.** You define a reference collection (typically your index documents, or an earlier production window) and a target collection (recent production queries). Both must be embeddings from the same encoder. Phoenix stores them with timestamps so 'last 7 days' vs 'first month after launch' is a one-click selection. **UMAP projection.** Phoenix projects both collections into 2D via UMAP (t-SNE is also available). The Embedding tab renders this as a scatter, color-coded by source. Drift shows up as the target color drifting away from the reference cluster, or as new sub-clusters in areas the reference does not cover. This is the human-readable signal that prompts the analyst to look closer. **Drift metrics.** Centroid distance (Euclidean between cluster centers), PSI (Population Stability Index), Wasserstein on per-dimension distributions. PSI above 0.25 is the conventional 'something shifted' threshold. The metric chart sits next to the UMAP scatter in the same tab. The valuable next step is the cluster sample label loop. Pick a drifted region on the scatter, sample queries from it, label them, and decide: index refresh (your corpus is stale), embedder retrain (your encoder no longer represents the new query types), or fallback policy (route those queries to a different pipeline).

TL;DR

Phoenix compares reference vs target embedding sets via UMAP projection plus a drift metric (PSI, Wasserstein, centroid distance) in the Embedding tab.

Memory aid

Easy to grasp

Picture comparing two big jars of marbles. One jar was sorted by color when you set up your shop; the other jar holds marbles customers brought in this month. You spread both jars out on a table by color and pattern. If the customer marbles pile up in a corner where you have nothing for sale, you have a mismatch and you need new stock. Phoenix does the same trick for query embeddings. It spreads them out on a 2D map alongside the documents you indexed. If the recent queries cluster in regions your documents do not cover, your knowledge base is out of date.

Key concepts

Observability and Tracing

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Embedding drift is the canonical 'silent quality degradation' in RAG systems. The retrieval API still returns chunks, the model still generates fluent answers, the trace still completes successfully. Nothing alerts. But the answers gradually become less helpful because user queries have drifted into regions your index does not cover well.

Phoenix is one of the few observability tools that makes this measurable out of the box. The UMAP-plus-PSI pattern it uses has become the de facto standard for this analysis, and the surrounding workflow (cluster, sample, label, remediate) is a senior-level pattern in itself.

The reference vs target primitive

Phoenix's drift tooling is built on comparing two embedding collections.

The reference is the baseline you are measuring against. Two common choices: the embeddings of your indexed documents (gives you 'do queries land where the corpus has coverage?'), or the embeddings of an earlier window of production queries (gives you 'have queries changed shape over time?'). Both have different interpretations; both are useful.

The target is the recent data. Typically a sliding window: last 7 days, last 30 days. Phoenix indexes embeddings with timestamps so windowing is a UI control.

Both collections must come from the same encoder. If the encoder version changes, the embeddings live in different spaces and comparison is meaningless. This is the single most important hygiene rule. Pin the encoder version in production and re-embed the reference set whenever you intentionally upgrade.

Phoenix supports both ad-hoc analysis (load two collections, compare) and scheduled drift monitoring (auto-recompute drift metrics on a daily basis with a rolling reference).

UMAP projection and what to look for

Quantitative drift metrics

The cluster sample label remediate loop

Three failure modes worth knowing

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Phoenix's Embedding tab with UMAP scatter and PSI chart is the canonical UI for this workflow.
Arize AX (the commercial sibling to Phoenix) adds production-scale drift monitoring on the same primitives.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you tell embedder drift apart from query-distribution drift?

Re-embed the reference set with the current encoder and compare; talk about version pinning and controlled re-embedding.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Looking at the drift number alone without sampling and labeling the drifted region. A score with no diagnosis tells you something changed but not what to do about it.

60 second bullets to scan on the way to the call.

Name the two embedding collections that feed Phoenix drift analysis
Identify which Phoenix UI tab surfaces the drift visualization

Primary sources. Browse if you want the original framing.

Arize Phoenix Documentation