When should automated eval escalate to human review in a production LLM product?

Short answer·Hard·4.0 · 0·~3 min·Asked atHebbia Mistral AI Robinhood·Relevant atAnthropic

Attempt it

You are running a production LLM assistant for a healthcare company. Describe which failure modes require escalation to human review versus which can be handled by automated eval, and explain the principle that distinguishes the two categories.

Free · 2 AI evals / day

In a production LLM product you cannot have a human read every response. You also cannot trust a machine to catch every dangerous one. So you split failures into two buckets using two questions: how bad is it if we miss this, and how reliably can software catch it? Automate the failures that are cheap to detect and low harm if missed. Format violations, wrong length, missing headers, off-topic answers with no harmful content, and factual slips on non-critical topics all fall here. A rule or an LLM-as-judge catches these at a near-zero miss rate, so a human adds nothing. Escalate the failures that are dangerous and hard to detect. In healthcare that means a wrong medication dose, unsafe medical advice, leaked patient details, or anything a user explicitly flags as a safety concern. Automated classifiers miss these too often because the harmful text looks syntactically normal. The single principle: escalate when severe harm meets unreliable detection. The healthcare context raises the harm side of that equation, so the escalation bar drops compared to a low-stakes consumer app. You feed every escalated incident back into your eval set so the same failure is caught next time.

The principle: escalate to human review when the failure carries severe harm AND your automated detector has an unreliable, high false-negative rate. Automate when harm is bounded and detection is accurate. The deciding axis is not the failure type. It is severity multiplied by detector unreliability. Escalate in healthcare. Potential medication dosage errors and dangerous advice slip past classifiers because the harmful text reads as syntactically normal. Indirect PII leakage evades detectors too. A response naming a March procedure at a named hospital leaks affiliation with no literal identifier. Outputs that already passed a safety classifier, by definition, need a human. So do any user-flagged safety reports. Automate the rest. Format violations yield to rule-based checks. Off-topic but harmless answers score reliably under an LLM-judge. Factual errors on non-personalized topics catch against a reference at scale. Healthcare shifts the harm term upward, so the escalation bar drops below a consumer app's. You also instrument production telemetry. Refusal rate, latency p99, thumbs-down rate, regeneration rate, and format-validation failures surface user-facing failures that a golden-set eval never modeled. Every escalated incident then becomes a new eval case. That closes the offline to online loop.

TL;DR

Escalate to human review when harm is severe AND detection is unreliable; automate when harm is bounded and the detector misses almost nothing. The deciding axis is the reliability gap, not the failure type.

Memory aid

Easy to grasp

Imagine an airport. The metal detector handles the easy, common, low-stakes stuff: it reliably catches keys and coins, so a machine is fine. But for something rare and dangerous that the scanner often misses, you call a human officer to inspect by hand. You do not send everyone to a human, because that does not scale, and you do not trust the machine for the scary cases it is bad at catching. A healthcare chatbot works the same way. A format slip or an off-topic answer is the keys and coins: cheap to catch, low harm if missed, so automate it. A wrong medication dose or a leaked patient detail is the dangerous case the scanner misses: high harm, unreliable detection, so a human must review it.

Key concepts

Online vs offline eval Human in the loop False-negative rate Production telemetry

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Automated eval is what lets an LLM product scale: a rule, an LLM-as-judge, or a safety classifier inspects outputs at a volume no human team could match. But automation has a blind spot, and in a high-stakes domain that blind spot can ship harm. The senior skill is knowing exactly where automation stops being trustworthy and a human must enter the loop.

This question asks you to draw that line for a healthcare assistant and, crucially, to state the principle behind it rather than recite a list. The principle is a two-axis risk surface: how severe is the harm if a failure ships, and how unreliable is your detector at catching it. The deep dive builds that surface, maps healthcare failures onto it, then extends past the offline golden set into the production telemetry and feedback loop that catch the failures you never anticipated.

The two-axis principle

Most candidates answer this question with a list of failure types and a gut call on each. That is the wrong frame. It collapses two separate questions into one and gives you no way to defend a borderline case. The right frame is a function over two independent variables.

The first axis is harm if missed: what does it cost if this failure reaches the user? The second axis is detector unreliability: the false-negative rate of your best automated check on this failure. These are independent. A failure can be catastrophic yet trivially detectable, or harmless yet hard to detect. Conflating them is the most common mistake, because it leads teams to escalate anything that sounds scary regardless of whether automation already catches it cleanly.

Escalate the quadrant where both are high: severe harm and an unreliable detector. Automate the quadrant where both are low: bounded harm and a near-zero false-negative rate. The remaining two quadrants are where cost and judgment live. Severe but reliably detected failures can often stay automated with a sampled human audit on the passes. Mild but hard to detect failures are usually not worth a human at all, since the expected harm does not justify the reviewer cost. Stating this surface, not the list, is what separates a staff answer from a senior one. It also gives you a principled way to defend why a given failure sits where it does.

Healthcare warps the harm axis

Why indirect PII evades automated detection

What to automate with confidence

Production telemetry and closing the loop

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Healthcare chatbots gated under HIPAA route any potential dosage or contraindication output to a clinician reviewer rather than a classifier alone.
LangSmith and Langfuse capture thumbs-down, regeneration, and latency p99 as live signals that feed curated traces back into the offline eval set.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you set the escalation threshold quantitatively rather than by intuition?

Estimate each failure mode's harm cost and its detector false-negative rate on a labelled holdout, then escalate where expected harm exceeds the reviewer cost per case. Recalibrate as detectors and traffic shift.

3 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Routing every flagged response to humans. That does not scale and trains reviewers to rubber-stamp. The right cut is severity times detector unreliability, not raw flag volume.

60 second bullets to scan on the way to the call.

The two-axis principle: harm severity times detector false-negative rate
Two healthcare failure modes that must escalate and why detection fails

Primary sources. Browse if you want the original framing.