**NeMo Guardrails** is NVIDIA's open-source framework for putting safety checks around an LLM application. It organises checks into five different rail types based on where in the request flow they run.
**Input rails** run first. They inspect the user's incoming message, flag jailbreak attempts, redact PII, refuse off-topic requests, before the model ever sees the input. Cheap and fast classifiers usually live here.
**Output rails** run after the model generates a response. They check the output for toxicity, sensitive data, hallucinated citations, or unsafe content before the user sees it.
**Retrieval rails** sit inside the RAG path. When the retriever returns documents, this rail can filter, redact, or sanitise them before they enter the model's context. Defends against indirect injection from poisoned documents.
**Dialog rails** shape the conversation itself. They say things like 'this assistant only answers questions about cooking' or 'if the user asks about medical dosing, hand off to a human.' They are about scope and flow, not just one message.
**Execution rails** wrap tool calls. Before the agent calls a tool, book a flight, send a refund, query a database, this rail runs pre and post checks: is this action allowed, are the arguments valid, do we need human confirmation?
Five moments, five rails. Stacking them together is how you build defence in depth.
**NeMo Guardrails** uses Colang, a small declarative language, to define rules that run at five distinct points in the request lifecycle. The point is defence in depth: no single check catches everything, so different rails address different risk classes at the moment they are best caught.
**Input rail**, runs on the raw user message. Typical checks: jailbreak detection, prompt-injection patterns, PII scrub, off-topic refusal. Cheap classifiers (Llama Prompt Guard, regex, a small NeMo model) live here.
**Output rail**, runs on the model's generated response. Typical checks: toxicity, hallucination patterns, PII leak, off-policy advice, structural validation against an expected schema. Heavier checks are fine here because output is on the critical path only once per turn.
**Retrieval rail**, runs on the retriever's results before they enter the LLM context. Typical checks: filter out documents the user is not authorised to see, scrub embedded instructions (indirect-injection defence), drop low-confidence retrievals. Treating retrieved content as untrusted is the central insight.
**Dialog rail**, runs at the conversation level, not one message. Defines `flows` in Colang: 'when user asks about competitors, deflect gracefully'; 'if conversation drifts to medical advice, hand off.' These rails encode product-policy decisions.
**Execution rail**, wraps tool calls. Before dispatch: validate arguments, check allowlist, require human confirmation for irreversible actions. After dispatch: validate the result before feeding it back into the model.
The rails compose. A single request typically traverses input rail, then dialog rail, then (if RAG) retrieval rail, then model, then output rail, then (if tool call) execution rail, then return.
Quick reference
TL;DR
NeMo Guardrails defines five rails, input, output, retrieval, dialog, execution, each running at a different point in the request path and addressing a distinct class of risk.
Picture a busy restaurant. There is a doorman who checks each customer at the entrance. There is a waiter who reviews each plate before it leaves the kitchen. The cookbook in the back has someone redacting confidential recipes before the chef reads them. The manager sets house rules about which dishes can be ordered together. And the cashier confirms before the bill goes to your card. The restaurant works smoothly because each person has one job at one specific moment. NeMo Guardrails uses five rails the same way: one for input, one for output, one for retrieval, one for dialog scope, and one for tool execution. The same rule can be enforced in different places, but each rail has its own moment to act.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Pure batch-mode LLM job with no user-facing surface and no tool calls Most rail types presuppose a live request-response loop; an offline batch generator typically needs only an output rail or none.
Latency budget under 200ms per turn Stacking five rails plus the model exceeds the budget; pick the two highest-value rails for your risk profile and skip the rest.
You need cross-cloud portability without NVIDIA tooling NeMo's tightest integration is with NIM; if you must run on AWS Bedrock or pure OSS, Guardrails AI or Cloud-native shields may be a better fit.
Quick scan · 2–4 min
Everything important, quickly.
6 min: the five-rail decomposition + what each rail inspects + why their moments are not interchangeable + retrieval rail as indirect-injection defence + dialog rails as deterministic policy + execution rails as LLM08 mitigation + composition and latency budgets.
Real-world examples
Real products, models, and research that use this idea.
1NVIDIA NIM bundles NeMo Guardrails with Colang flows pre-baked for common safety policies in 2026.
2Bedrock Guardrails on AWS mirrors the input-output split; NeMo's five-rail model is the more granular reference.
3Cisco AI Defense and OpenAI's moderation stack both adopt input/output rail separation explicitly.
4Anthropic's Claude documentation recommends a similar layered architecture: input filter, model, output filter, with retrieval sanitisation as a separate concern.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do dialog rails differ from carefully written system prompts?
Sign in or go Pro to see the approach for this follow-up.
QHow does the retrieval rail interact with the dual-LLM pattern?
Sign in or go Pro to see the approach for this follow-up.
QWhat is the right latency budget per rail in a production stack?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating all rails as interchangeable input-output filters. Each rail sits at a specific moment of the request flow and addresses risks the others cannot reach.