When should automated eval escalate to human review in a production LLM product?

In a production LLM product you cannot have a human read every response. You also cannot trust a machine to catch every dangerous one. So you split failures into two buckets using two questions: how bad is it if we miss this, and how reliably can software catch it? Automate the failures that are cheap to detect and low harm if missed. Format violations, wrong length, missing headers, off-topic answers with no harmful content, and factual slips on non-critical topics all fall here. A rule or an LLM-as-judge catches these at a near-zero miss rate, so a human adds nothing. Escalate the failures that are dangerous and hard to detect. In healthcare that means a wrong medication dose, unsafe medical advice, leaked patient details, or anything a user explicitly flags as a safety concern. Automated classifiers miss these too often because the harmful text looks syntactically normal. The single principle: escalate when severe harm meets unreliable detection. The healthcare context raises the harm side of that equation, so the escalation bar drops compared to a low-stakes consumer app. You feed every escalated incident back into your eval set so the same failure is caught next time.

The principle: escalate to human review when the failure carries severe harm AND your automated detector has an unreliable, high false-negative rate. Automate when harm is bounded and detection is accurate. The deciding axis is not the failure type. It is severity multiplied by detector unreliability. Escalate in healthcare. Potential medication dosage errors and dangerous advice slip past classifiers because the harmful text reads as syntactically normal. Indirect PII leakage evades detectors too. A response naming a March procedure at a named hospital leaks affiliation with no literal identifier. Outputs that already passed a safety classifier, by definition, need a human. So do any user-flagged safety reports. Automate the rest. Format violations yield to rule-based checks. Off-topic but harmless answers score reliably under an LLM-judge. Factual errors on non-personalized topics catch against a reference at scale. Healthcare shifts the harm term upward, so the escalation bar drops below a consumer app's. You also instrument production telemetry. Refusal rate, latency p99, thumbs-down rate, regeneration rate, and format-validation failures surface user-facing failures that a golden-set eval never modeled. Every escalated incident then becomes a new eval case. That closes the offline to online loop.