The PM asks how to tell if the chatbot got better after the last deploy. Walk through setting up an A/B test.
Split traffic at the session level, measure quality via LLM-judge plus user signals, run until statistically significant, and set auto-rollback guardrails on safety metrics.
Imagine you own two restaurants and want to know which chef cooks better. You randomly assign each customer to one restaurant for their entire meal, not switching chefs between courses. You track whether they finish the meal, leave a good review, and how long the food takes. You also have a rule: if more than a few customers get sick, you shut that kitchen down immediately. An A/B test for a chatbot works the same way. Each user gets one version for their whole conversation, you measure quality and speed, and you have a safety shutoff.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
A/B testing is the gold standard for answering the question "did the chatbot actually get better?" But applying it to an LLM product is not the same as testing a new button color on a web page. LLM conversations are multi-turn, quality is hard to measure at scale, and failures can be catastrophic rather than merely annoying.
This explanation walks through the full setup: randomization, metrics, duration, guardrails, and reporting. Each section highlights where the LLM domain departs from standard web experimentation and what engineering choices those departures require.
Session-level randomization and the carryover problem
Standard web A/B tests randomize at the page view or request level. For an LLM chatbot, this breaks immediately. A multi-turn conversation carries context forward: the model's response on turn 1 becomes part of the prompt on turn 2. If a user hits Model A on turn 1 and Model B on turn 2, Model B is responding to context it did not generate. The user's experience reflects neither model cleanly.
The fix is session-level randomization. Hash the session ID to a bucket (Control or Treatment) at session start and stick it for the session's lifetime. Every turn in the session hits the same model. For stateless APIs where sessions are not explicit, attach the bucket to the user ID or API key.
A subtlety: if users can start multiple sessions per day, cross-session contamination is possible (the user's expectations shift after experiencing the other variant). This is generally acceptable because each session is measured independently, but if you suspect strong learning effects, randomize at the user level instead of the session level.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Anthropic uses internal A/B testing with session-level bucketing and LLM-as-judge quality scoring to validate Claude model improvements before production promotion.
- OpenAI's ChatGPT runs A/B experiments on prompt and model changes with session-sticky randomization and monitors safety metrics in real time.
What an interviewer would ask next. Try answering before peeking at the approach.
QYour chatbot has 10,000 daily sessions but only 2% of users click thumbs-up or thumbs-down. How do you get enough quality signal to power the A/B test?
Supplement sparse explicit feedback with implicit signals: session length, follow-up question count, task completion rate. Run LLM-as-judge on a larger sample (10 percent instead of 5 percent) to get a continuous quality score on every sampled conversation. The judge score is the primary metric; thumbs-up rate is a secondary confirmation signal.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Randomizing at the request level instead of the session level, so multi-turn conversations get contaminated by mid-conversation model switches.
60 second bullets to scan on the way to the call.
Explain why randomization must happen at the session level for multi-turn products.
Name three quality metrics appropriate for an LLM A/B test.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.