Which considerations are uniquely important for A/B testing LLM systems vs standard UI testing?

Attempt it

Quick reference

TL;DR

LLM A/B tests differ on four real axes: indirect quality metrics, user-level assignment for carryover, higher variance needing larger samples, and safety guardrails. The 7-day rule and bare p-value are generic.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine testing two waiters in a restaurant instead of two buttons on a website. A button click is easy to count, but waiter quality is fuzzy: was the answer helpful, polite, correct? That is harder to score, so you need more diners before you trust the result. You also cannot swap waiters mid-meal for one table, or the conversation gets confused, so you give a whole table to one waiter for the visit. And because a waiter could say something genuinely harmful, you keep a manager ready to pull them off the floor instantly. Some habits are the same for any test: run long enough to see weekday versus weekend patterns, and be careful with statistics when you check many things at once. Those last two are good practice everywhere, not special to AI.

Key concepts

LLM evaluation LLM system design Retrieval-augmented generation

Quick scan · 2–4 min

Everything important, quickly.

5 min: four real LLM-specific axes (indirect metrics, user-level assignment, variance and power, safety guardrails) plus the two generic distractors, then the production setup of randomization unit, sample sizing, and sequential guardrails.

Side-by-side

Design choice	Standard UI A/B test	LLM A/B test
Primary metric	Direct event (click, conversion)	Indirect proxy (judge score, follow-up rate)
Randomization unit	Request or user, often interchangeable	User or session, to avoid multi-turn carryover
Signal variance	Low, bounded Bernoulli	High, drives larger samples and detectable effect
Failure handling	Metric regression, reversible	Possible user harm, needs auto-stop guardrails
Run length and p-value	Capture weekly seasonality, correct for multiple looks	Same generic rigor, not LLM-specific

Real-world examples

Real products, models, and research that use this idea.

Statsig and LaunchDarkly ship user-level assignment plus sequential testing, the default setup teams use to A/B test LLM features without carryover.
GitHub Copilot rolled out model changes behind staged A/B tests gated on acceptance rate as primary and latency as a guardrail metric.
OpenAI and Anthropic run online A/B tests on model updates with refusal-rate and safety-flag guardrails wired to auto-rollback.
LangSmith and Statsig integrate online LLM metrics (thumbs, follow-up rate) directly into experiment dashboards for live quality tracking.
Eppo and GrowthBook expose CUPED variance reduction, which teams use to claw back power on noisy LLM quality metrics.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you size an LLM A/B test when the primary metric is a noisy 1-to-5 judge score?

Sign in or go Pro to see the approach for this follow-up.

QWhy does request-level randomisation break a multi-turn conversational eval?

Sign in or go Pro to see the approach for this follow-up.

QHow do you peek at an LLM A/B test daily without inflating false positives?

Sign in or go Pro to see the approach for this follow-up.

QWhat guardrail metrics would you wire to auto-rollback for a chat assistant challenger?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.