LLM A/B tests differ on four real axes: indirect quality metrics, user-level assignment for carryover, higher variance needing larger samples, and safety guardrails. The 7-day rule and bare p-value are generic.
Imagine testing two waiters in a restaurant instead of two buttons on a website. A button click is easy to count, but waiter quality is fuzzy: was the answer helpful, polite, correct? That is harder to score, so you need more diners before you trust the result. You also cannot swap waiters mid-meal for one table, or the conversation gets confused, so you give a whole table to one waiter for the visit. And because a waiter could say something genuinely harmful, you keep a manager ready to pull them off the floor instantly. Some habits are the same for any test: run long enough to see weekday versus weekend patterns, and be careful with statistics when you check many things at once. Those last two are good practice everywhere, not special to AI.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A/B testing for LLMs is online controlled experimentation applied to a model or prompt change: split live traffic, expose one arm to the challenger, and compare outcomes against the control. The statistical machinery is the same one web teams have used for two decades. What changes is the nature of the thing you are measuring and the harm a bad arm can cause.
This question presents six considerations and asks which are uniquely important for LLMs. Four are real (indirect quality metrics, multi-turn carryover, higher variance demanding larger samples, and catastrophic safety guardrails). Two are distractors that dress up generic experimentation rigor (the 7-day window and a bare p-value below 0.05) as LLM-specific. The deep dive works through each, then assembles the production setup an interviewer wants to hear: randomization unit, metric stack, power, and guardrails.
Indirect quality metrics: the measurement problem
In a UI test the outcome is usually a clean event: a click, a purchase, a sign-up. You count it and you are done. LLM quality has no such clean signal. Helpfulness, faithfulness, and relevance are latent qualities you must operationalise into something countable.
Teams do this three ways, and each has a cost. Implicit signals (thumbs up rate, follow-up question rate, session length, copy of answer rate) are cheap and online but only loosely correlated with true quality. LLM-as-judge scores are richer but carry their own bias and per-call expense. Human ratings are the gold standard but slow and small.
The practical consequence for the experiment is that your primary metric is a proxy, and a noisy one. You must choose it before traffic flows, decide how to aggregate it (mean score, win rate, fraction above a threshold), and accept that it tracks the thing you care about imperfectly. This is the root cause of the variance problem that follows.
There is also a definitional trap. Quality is multi-dimensional, so a single scalar hides tradeoffs: a challenger can be more helpful but less faithful, or more concise but more likely to refuse. Picking one headline metric to power the experiment forces you to be explicit about what you are optimising, and to push the other dimensions into the guardrail set rather than pretending one number captures everything.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Design choice | Standard UI A/B test | LLM A/B test |
|---|---|---|
| Primary metric | Direct event (click, conversion) | Indirect proxy (judge score, follow-up rate) |
| Randomization unit | Request or user, often interchangeable | User or session, to avoid multi-turn carryover |
| Signal variance | Low, bounded Bernoulli | High, drives larger samples and detectable effect |
| Failure handling | Metric regression, reversible | Possible user harm, needs auto-stop guardrails |
| Run length and p-value | Capture weekly seasonality, correct for multiple looks | Same generic rigor, not LLM-specific |
Real products, models, and research that use this idea.
- Statsig and LaunchDarkly ship user-level assignment plus sequential testing, the default setup teams use to A/B test LLM features without carryover.
- GitHub Copilot rolled out model changes behind staged A/B tests gated on acceptance rate as primary and latency as a guardrail metric.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you size an LLM A/B test when the primary metric is a noisy 1-to-5 judge score?
Estimate the metric variance from a pilot, fix the minimum detectable effect you care about, then use sample size proportional to variance over the squared effect. Consider CUPED or stratification to cut variance and reclaim power before adding more traffic.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating an LLM A/B test like a button-color test. Quality is a noisy proxy, conversations carry over across turns, and a bad output can harm a user instantly, none of which a click-rate test must handle.
60 second bullets to scan on the way to the call.
Why the randomization unit must be user or session, not request
How multi-turn carryover biases the estimated effect
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.