You run a customer-support agent that can issue refunds and update accounts. Outline a tiered escalation path for guardrail trips, what happens at each tier, who is paged, and what state is preserved.
Free · 2 AI evals / day
Different guardrail trips deserve different responses. A low-confidence classifier ping is not the same incident as a vetoed refund. The design question is how to map each trip type to a paging path and a state-preservation contract.
**Tier 1, soft trip.** Low-confidence classifier hit, output rail rewrote part of a response, off-topic gate fired ambiguously. The user sees a polite fallback or a slightly modified answer. The event is logged with full context, input, retrieved documents, model output, classifier scores, for offline review. No human paged. The review queue is sampled weekly for drift detection.
**Tier 2, hard trip.** High-confidence injection attempt, PII leak attempt, or policy violation in output. The conversation is short-circuited with a templated apology. State (conversation id, user id, agent action queue, last tool call, classifier verdict) is preserved in an audit table. The customer-success rota is notified asynchronously, they get to the case within hours, not minutes.
**Tier 3, irreversible action blocked.** The agent tried to issue a refund, modify an account, or call an external API and a guardrail vetoed. The on-call SRE is paged in real time. A SEV-3 incident is opened. The user is offered a callback. Preserved state includes the exact tool call that was blocked and the policy rule that fired, so the human can approve, deny, or modify.
**Three tiers, mapped to blast radius, each with a defined user message and state contract.**
Tier 1 is the silent-log path. A soft trip, low-confidence classifier, output-rail rewrite, off-topic ambiguity, does not interrupt the user. The response is delivered (or politely fallback-routed if the trip warranted modification) and the event lands in a review queue with full context: input, retrieved documents, model output, classifier scores, model version. No one is paged. The queue is sampled weekly against a golden set to drive classifier tuning.
Tier 2 is the async-review path. A hard trip, high-confidence injection, PII leak attempt, policy violation, short-circuits the conversation with a templated apology that names the policy at a high level ('I cannot help with that and have logged this conversation for review'). The audit table captures conversation id, user id, the full transcript, the agent's pending action queue, the classifier verdict, and any tool calls already attempted. The customer-success rota is notified through a normal-priority channel and picks up the case within their stated SLA, typically hours.
Tier 3 is the synchronous-page path. The agent attempted an irreversible action, refund, account modification, external API call, and a guardrail blocked it. On-call SRE is paged immediately. A SEV-3 incident opens. The user gets an immediate templated message offering a callback. Preserved state includes the precise tool call (function name, arguments, intended downstream effect) and the policy rule that fired, so the human can approve the call as-is, modify it, or deny it.
The key invariant: tier is determined by the blast radius of the attempted action and the confidence of the guardrail, not by the classifier score alone.
Quick reference
TL;DR
Three tiers, silent log, async review with state snapshot, and synchronous page on a blocked irreversible action, each with a defined user-facing message and a preserved-state contract.
Imagine a security guard at a bank lobby. If someone walks in muttering oddly, the guard takes a note for later review. If someone tries to push past the counter, the guard intervenes and calls the manager when there is time. If someone reaches over the counter for the cash drawer, the guard hits the alarm immediately and the police come. Same job, three different intensities matched to how bad the worst case is. A customer-support agent's guardrails work the same way, a low-confidence classifier ping is the muttering, an injection attempt is the pushy customer, and a vetoed refund is the reach for the cash drawer.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
The agent has no side effects and only emits text, read-only retrieval or pure summarisation Without irreversible actions, the tier-3 path has nothing to gate; collapse to tier 1 (log) and tier 2 (async review) only
There is no human team available to receive escalations, even asynchronously A tier-2 queue with no one to drain it is just a denial of service to affected users; redesign the product so guardrail trips end the conversation cleanly instead of staging a handoff that never happens
Volume is so low that every event can be reviewed manually Tier discipline adds overhead that pays off at scale; at very low volume, single-queue review is operationally simpler
Quick scan · 2–4 min
Everything important, quickly.
9 min: define the three tiers, walk the blast-radius × confidence matrix, set the state contract per tier, draft the user-facing message templates, and close on tier-rate calibration and feedback loops.
Refund / account modification / external API blocked
Templated message with callback offer
Synchronous SRE page, SEV-3 incident
Real-world examples
Real products, models, and research that use this idea.
1Klarna's customer-support agent (2026) routes irreversible-action blocks to a synchronous human handoff and silent-logs sub-threshold safety classifier hits.
2Anthropic's Claude in enterprise deployments through AWS Bedrock 2026 documents tiered escalation as a recommended pattern alongside Bedrock Guardrails.
3Microsoft Copilot for Service uses a tiered model where soft trips never page, hard trips route to a queue, and irreversible-action trips short-circuit to a human agent.
4Datadog LLM Observability 2026 ships built-in tier dashboards for guardrail events with SLA tracking per tier.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you tune the tier-2 vs tier-3 threshold over time?
Sign in or go Pro to see the approach for this follow-up.
QWhat if your customer-success team is overwhelmed and the tier-2 queue is growing faster than they can drain it?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Paging humans on every classifier ping. The team gets alert fatigue, ignores the queue, and misses the real incidents.