Design an escalation path for guardrail trips in a customer facing agent

Short answer·Medium·4.0 · 0·~3 min·Asked atGoldman Sachs Stability Ai Tencent

Attempt it

You run a customer support agent that can issue refunds and update accounts. Outline a tiered escalation path for guardrail trips, what happens at each tier, who is paged, and what state is preserved.

Free · 2 AI evals / day

Different guardrail trips deserve different responses. A low-confidence classifier ping is not the same incident as a vetoed refund. The design question is how to map each trip type to a paging path and a state-preservation contract. **Tier 1, soft trip.** Low-confidence classifier hit, output rail rewrote part of a response, off-topic gate fired ambiguously. The user sees a polite fallback or a slightly modified answer. The event is logged with full context, input, retrieved documents, model output, classifier scores, for offline review. No human paged. The review queue is sampled weekly for drift detection. **Tier 2, hard trip.** High-confidence injection attempt, PII leak attempt, or policy violation in output. The conversation is short-circuited with a templated apology. State (conversation id, user id, agent action queue, last tool call, classifier verdict) is preserved in an audit table. The customer-success rota is notified asynchronously, they get to the case within hours, not minutes. **Tier 3, irreversible action blocked.** The agent tried to issue a refund, modify an account, or call an external API and a guardrail vetoed. The on-call SRE is paged in real time. A SEV-3 incident is opened. The user is offered a callback. Preserved state includes the exact tool call that was blocked and the policy rule that fired, so the human can approve, deny, or modify.

**Three tiers, mapped to blast radius, each with a defined user message and state contract.** Tier 1 is the silent-log path. A soft trip, low-confidence classifier, output-rail rewrite, off-topic ambiguity, does not interrupt the user. The response is delivered (or politely fallback-routed if the trip warranted modification) and the event lands in a review queue with full context: input, retrieved documents, model output, classifier scores, model version. No one is paged. The queue is sampled weekly against a golden set to drive classifier tuning. Tier 2 is the async-review path. A hard trip, high-confidence injection, PII leak attempt, policy violation, short-circuits the conversation with a templated apology that names the policy at a high level ('I cannot help with that and have logged this conversation for review'). The audit table captures conversation id, user id, the full transcript, the agent's pending action queue, the classifier verdict, and any tool calls already attempted. The customer-success rota is notified through a normal-priority channel and picks up the case within their stated SLA, typically hours. Tier 3 is the synchronous-page path. The agent attempted an irreversible action, refund, account modification, external API call, and a guardrail blocked it. On-call SRE is paged immediately. A SEV-3 incident opens. The user gets an immediate templated message offering a callback. Preserved state includes the precise tool call (function name, arguments, intended downstream effect) and the policy rule that fired, so the human can approve the call as-is, modify it, or deny it. The key invariant: tier is determined by the blast radius of the attempted action and the confidence of the guardrail, not by the classifier score alone.

TL;DR

Three tiers, silent log, async review with state snapshot, and synchronous page on a blocked irreversible action, each with a defined user-facing message and a preserved-state contract.

Memory aid

Easy to grasp

Imagine a security guard at a bank lobby. If someone walks in muttering oddly, the guard takes a note for later review. If someone tries to push past the counter, the guard intervenes and calls the manager when there is time. If someone reaches over the counter for the cash drawer, the guard hits the alarm immediately and the police come. Same job, three different intensities matched to how bad the worst case is. A customer-support agent's guardrails work the same way, a low-confidence classifier ping is the muttering, an injection attempt is the pushy customer, and a vetoed refund is the reach for the cash drawer.

Key concepts

Safety and Guardrails Agents

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Guardrails fire at different intensities for different reasons. A low-confidence classifier ping on a read-only response is a different operational event from a high-confidence policy block on a refund attempt. Treating them identically, either by paging on every event or by silent-logging everything, fails in opposite directions.

The production answer is a tiered escalation system where each tier has a defined trigger, a defined user-facing behaviour, a defined state-preservation contract, and a defined human-notification path. Getting the tier mapping right is what makes the guardrail system usable rather than alarm spam or silent neglect.

The two-dimensional tier matrix

The natural reflex is to tier on classifier confidence alone, high confidence equals high tier. This is wrong. A high-confidence classifier hit on a benign-action surface is still a low-impact event; a low-confidence hit on a refund attempt is potentially a high-impact event.

The correct tier signal is two-dimensional: the blast radius of the attempted action crossed with the confidence of the trip. Blast radius ranges from zero (read-only text output) through low (account preference change) up to high (refund, account modification, external API call). Confidence is the classifier or rule's certainty that something is wrong.

The matrix gives three operational tiers. Low-radius × any confidence = tier 1. Medium-radius × medium or high confidence, or low-radius × very-high confidence = tier 2. High-radius × any actionable confidence = tier 3. The fact that an irreversible-action block can land in tier 3 even at moderate confidence reflects that the cost of a wrong action is asymmetric, the false-negative cost dominates the false-positive cost when refunds are involved.

Tier 1: silent log, sampled review

Tier 2: short-circuit, async handoff

Tier 3: synchronous page, real-time handoff

Calibrating rates, closing the loop, evolving the matrix

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Tier	Trigger	User-facing action	Human notification
Tier 1 (soft)	Low-confidence classifier hit, output rail rewrite	Polite fallback or modified response	None; sampled review queue
Tier 2 (hard)	High-confidence injection, PII leak attempt, policy violation	Templated apology, conversation short-circuited	Async customer-success queue, hours SLA
Tier 3 (irreversible action)	Refund / account modification / external API blocked	Templated message with callback offer	Synchronous SRE page, SEV-3 incident

Real products, models, and research that use this idea.

Klarna's customer-support agent (2026) routes irreversible-action blocks to a synchronous human handoff and silent-logs sub-threshold safety classifier hits.
Anthropic's Claude in enterprise deployments through AWS Bedrock 2026 documents tiered escalation as a recommended pattern alongside Bedrock Guardrails.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you tune the tier-2 vs tier-3 threshold over time?

Talk about a weekly review of the tier-3 page rate vs on-call capacity, a monthly review of incidents that should have been tier-3 but were tier-2, and a quarterly recalibration as the agent's tool surface evolves.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Paging humans on every classifier ping. The team gets alert fatigue, ignores the queue, and misses the real incidents.

60 second bullets to scan on the way to the call.

Map tier to a combination of blast radius and trip confidence
Define the state contract preserved at each tier

Primary sources. Browse if you want the original framing.