You are running a production LLM assistant for a healthcare company. Describe which failure modes require escalation to human review versus which can be handled by automated eval, and explain the principle that distinguishes the two categories.
Free · 2 AI evals / day
In a production LLM product you cannot have a human read every response. You also cannot trust a machine to catch every dangerous one. So you split failures into two buckets using two questions: how bad is it if we miss this, and how reliably can software catch it?
Automate the failures that are cheap to detect and low harm if missed. Format violations, wrong length, missing headers, off-topic answers with no harmful content, and factual slips on non-critical topics all fall here. A rule or an LLM-as-judge catches these at a near-zero miss rate, so a human adds nothing.
Escalate the failures that are dangerous and hard to detect. In healthcare that means a wrong medication dose, unsafe medical advice, leaked patient details, or anything a user explicitly flags as a safety concern. Automated classifiers miss these too often because the harmful text looks syntactically normal.
The single principle: escalate when severe harm meets unreliable detection. The healthcare context raises the harm side of that equation, so the escalation bar drops compared to a low-stakes consumer app. You feed every escalated incident back into your eval set so the same failure is caught next time.
The principle: escalate to human review when the failure carries severe harm AND your automated detector has an unreliable, high false-negative rate. Automate when harm is bounded and detection is accurate.
The deciding axis is not the failure type. It is severity multiplied by detector unreliability.
Escalate in healthcare. Potential medication dosage errors and dangerous advice slip past classifiers because the harmful text reads as syntactically normal. Indirect PII leakage evades detectors too. A response naming a March procedure at a named hospital leaks affiliation with no literal identifier. Outputs that already passed a safety classifier, by definition, need a human. So do any user-flagged safety reports.
Automate the rest. Format violations yield to rule-based checks. Off-topic but harmless answers score reliably under an LLM-judge. Factual errors on non-personalized topics catch against a reference at scale.
Healthcare shifts the harm term upward, so the escalation bar drops below a consumer app's. You also instrument production telemetry. Refusal rate, latency p99, thumbs-down rate, regeneration rate, and format-validation failures surface user-facing failures that a golden-set eval never modeled. Every escalated incident then becomes a new eval case. That closes the offline to online loop.
Quick reference
TL;DR
Escalate to human review when harm is severe AND detection is unreliable; automate when harm is bounded and the detector misses almost nothing. The deciding axis is the reliability gap, not the failure type.
Imagine an airport. The metal detector handles the easy, common, low-stakes stuff: it reliably catches keys and coins, so a machine is fine. But for something rare and dangerous that the scanner often misses, you call a human officer to inspect by hand. You do not send everyone to a human, because that does not scale, and you do not trust the machine for the scary cases it is bad at catching. A healthcare chatbot works the same way. A format slip or an off-topic answer is the keys and coins: cheap to catch, low harm if missed, so automate it. A wrong medication dose or a leaked patient detail is the dangerous case the scanner misses: high harm, unreliable detection, so a human must review it.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
A low-stakes consumer product where no output can cause physical or legal harm The harm axis is flat, so blanket human review is wasted cost. Automate broadly and sample for audit.
Failures with a near-zero false-negative rate like schema or format validation A deterministic check already catches these reliably, so human review adds latency and no accuracy.
Reviewer capacity is already saturated and escalation volume keeps rising Adding more escalation categories degrades into rubber-stamping. Recalibrate the cut before widening the net.
Real-time latency-critical paths where synchronous human review is impossible Human review cannot sit inline. Use an automated gate plus asynchronous sampled audit instead.
Quick scan · 2–4 min
Everything important, quickly.
6 min: the two-axis principle (harm times detector unreliability), the healthcare escalate list, the automate list, production telemetry signals, and closing the offline to online loop by feeding incidents back into eval.
Real-world examples
Real products, models, and research that use this idea.
1Healthcare chatbots gated under HIPAA route any potential dosage or contraindication output to a clinician reviewer rather than a classifier alone.
2LangSmith and Langfuse capture thumbs-down, regeneration, and latency p99 as live signals that feed curated traces back into the offline eval set.
3Microsoft Azure AI Content Safety and Guardrails AI run as automated gates for format and policy, with sampled human audit on the passes.
4Patronus AI and Galileo market production monitoring that flags drift via refusal-rate and hallucination signals for human triage.
5Modern frontier assistants (Claude Opus 4.7, GPT-5.5) expose safety-classifier scores that teams threshold for escalation rather than treat as final.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you set the escalation threshold quantitatively rather than by intuition?
Sign in or go Pro to see the approach for this follow-up.
QHow do you stop the escalation channel from decaying into rubber-stamping over time?
Sign in or go Pro to see the approach for this follow-up.
QWhat live telemetry signals catch user-facing failures that a golden-set eval misses entirely?
Sign in or go Pro to see the approach for this follow-up.
QHow does the offline to online feedback loop actually get closed in practice?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Routing every flagged response to humans. That does not scale and trains reviewers to rubber-stamp. The right cut is severity times detector unreliability, not raw flag volume.