Drag each answer to line up with its matching prompt
Input rail
Wraps an action (tool call) with pre and post checks before the runtime dispatches it
Output rail
Inspects the model's generated response before it reaches the user or downstream system
Retrieval rail
Inspects and possibly rewrites or blocks user supplied input before any model call
Dialog rail
Constrains the conversational flow itself, what topics are in scope, when to hand off to a human
Execution rail
Filters or sanitises documents returned from the retriever before they enter the LLM context
NeMo Guardrails defines five rails, input, output, retrieval, dialog, execution, each running at a different point in the request path and addressing a distinct class of risk.
Picture a busy restaurant. There is a doorman who checks each customer at the entrance. There is a waiter who reviews each plate before it leaves the kitchen. The cookbook in the back has someone redacting confidential recipes before the chef reads them. The manager sets house rules about which dishes can be ordered together. And the cashier confirms before the bill goes to your card. The restaurant works smoothly because each person has one job at one specific moment. NeMo Guardrails uses five rails the same way: one for input, one for output, one for retrieval, one for dialog scope, and one for tool execution. The same rule can be enforced in different places, but each rail has its own moment to act.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
NeMo Guardrails is NVIDIA's open-source framework for adding policy and safety logic around a production LLM application. Its central architectural choice is to decompose 'guardrails' into five distinct rail types based on where they run in the request lifecycle. This decomposition matters because different risks are addressable only at specific moments, and conflating all guardrails into a single input-output filter leaves the largest attack surfaces, retrieval and execution, undefended.
This walkthrough takes each rail in turn: what it inspects, where it runs in the flow, what risks it owns, and what it cannot reach. By the end you should be able to look at a production agent stack, name the rails you have, and identify the rails you are missing.
Mental model: the five rails are not five flavours of the same filter. They are five different checkpoints in a request's journey, each one closing off a class of failure the others cannot see.
Input and output rails: the request bookends
Input rail
The input rail runs first, on the raw user message before any model call. It is the cheapest place to catch the obvious: known jailbreak patterns, PII the user accidentally pasted, off-topic prompts, language outside supported set. Tools that live here are small and fast, Llama Prompt Guard for injection detection, regex or Microsoft Presidio for PII, a small NeMo classifier for off-topic.
The latency budget is tight, usually 50 to 100ms, because every request pays it. Avoid running heavy LLM-as-judge checks in the input rail; that doubles user-perceived latency for an attack class that small classifiers already handle.
Output rail
The output rail runs after the model generates, before the response reaches the user. This is the place for toxicity classifiers, structural validation against an expected schema, citation-required checks, and hallucination grounding (does this claim appear in the retrieved context?).
A common policy split: input rail fails fast and loudly (return a templated refusal); output rail can fail closed on safety-critical routes (block the response) or fail open on low-stakes routes (release with a flag). The right choice is per-route policy, not a global flag.
What input and output rails cannot do
They cannot see retrieved documents (those flow into the model context after the input rail), they cannot prevent a tool call (the execution rail owns that), and they cannot shape multi-turn dialog (the dialog rail owns that). Trying to encode policy in either rail alone collapses the abstraction NeMo gave you.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- NVIDIA NIM bundles NeMo Guardrails with Colang flows pre-baked for common safety policies in 2026.
- Bedrock Guardrails on AWS mirrors the input-output split; NeMo's five-rail model is the more granular reference.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do dialog rails differ from carefully written system prompts?
Dialog rails are Colang flows that run deterministically outside the model loop; jailbreaks cannot override them. System prompts are tokens the model reads and can be persuaded around.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating all rails as interchangeable input-output filters. Each rail sits at a specific moment of the request flow and addresses risks the others cannot reach.
60 second bullets to scan on the way to the call.
The five NeMo rail types by name
What each rail inspects and at which moment
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.