The PM asks how to tell if the chatbot got better after the last deploy. Walk through setting up an A/B test.

Attempt it

Free · 2 AI evals / day

Quick reference

TL;DR

Split traffic at the session level, measure quality via LLM-judge plus user signals, run until statistically significant, and set auto-rollback guardrails on safety metrics.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine you own two restaurants and want to know which chef cooks better. You randomly assign each customer to one restaurant for their entire meal, not switching chefs between courses. You track whether they finish the meal, leave a good review, and how long the food takes. You also have a rule: if more than a few customers get sick, you shut that kitchen down immediately. An A/B test for a chatbot works the same way. Each user gets one version for their whole conversation, you measure quality and speed, and you have a safety shutoff.

Key concepts

A/B Testing

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

Start with session-level randomization and explain the carryover contamination problem. Walk the three metric tiers: quality (LLM-as-judge, user signals), operational (latency, cost), safety (refusal and flag rates). Explain auto-rollback guardrails and why they are non-negotiable for LLM products. Cover sequential testing for duration. Close with reporting CIs for all metrics and framing the quality vs latency tradeoff as a product decision.

Real-world examples

Real products, models, and research that use this idea.

Anthropic uses internal A/B testing with session-level bucketing and LLM-as-judge quality scoring to validate Claude model improvements before production promotion.
OpenAI's ChatGPT runs A/B experiments on prompt and model changes with session-sticky randomization and monitors safety metrics in real time.
Google's Bard (now Gemini) evaluation pipeline includes online A/B testing with automatic experiment halts when safety metrics breach thresholds.
Spotify and Netflix use session-level A/B testing for recommendation models with sequential testing frameworks, applying the same principle to LLM-powered features.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QYour chatbot has 10,000 daily sessions but only 2% of users click thumbs-up or thumbs-down. How do you get enough quality signal to power the A/B test?

Sign in or go Pro to see the approach for this follow-up.

QYour A/B test shows a significant quality improvement but a 30% latency regression. How do you present this to the PM?

Sign in or go Pro to see the approach for this follow-up.

QHow would you adapt this A/B test design for a single-turn API product with no sessions?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Randomizing at the request level instead of the session level, so multi-turn conversations get contaminated by mid-conversation model switches.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.