A RAG pipeline fetches documents and pastes them into the prompt, but pasting context in is not enough. The system prompt has to tell the model how to use that context. Three directives form the canonical **grounding** set.
First, ground only: answer using just the provided context, not training data knowledge. Without this, the model treats the chunks as a hint and blends in half remembered facts, which reintroduces hallucination.
Second, cite chunks: tag each claim with its source chunk number, like [3]. This makes answers verifiable for the user and lets an eval harness check attribution automatically.
Third, refuse on insufficient context: if the chunks do not contain the answer, say so instead of guessing. This is the safety valve for the common case where retrieval misses.
The other options, a friendly tone, putting chunks in the user message, or restricting topics, are real prompt-engineering choices, but none of them changes whether the model stays grounded. That is why only the first three are correct.
Retrieval only solves half the problem. Once the chunks are in the prompt, the system prompt has to constrain how the model uses them. Three directives do exactly that.
**Ground-only** tells the model to answer from the provided context and not fall back on parametric knowledge. Skip it and the model treats context as a suggestion, blending in training data facts and reintroducing the hallucination RAG was meant to remove.
Cite-chunks requires each claim to carry its source chunk number, like [3]. This buys you verifiable, attributable answers and, just as important, gives an eval harness a machine-checkable signal for faithfulness.
Refuse-if-insufficient is the safety valve: when retrieval misses and the chunks do not contain the answer, the model must say so rather than fabricate. This is the single most common production failure, the model confidently answering from irrelevant context.
The distractors are all real techniques addressing different concerns. Tone is a UX choice orthogonal to faithfulness. Whether chunks live in the system or user message is a minor placement detail modern models handle either way. Domain restriction governs scope, not grounding. None of the three changes whether the answer is grounded in the retrieved evidence.
Quick reference
TL;DR
The grounding triad for a RAG system prompt: answer from context only, cite chunks by number, and refuse when context is insufficient. Tone and chunk placement are unrelated.
Imagine a librarian who hands you a stack of pages and asks you to answer a customer's question. You give the librarian three rules. First: answer only from these pages, do not use what you already know. Second: after each sentence, write which page it came from so anyone can check. Third: if the pages do not actually contain the answer, just say so instead of making something up. Those three rules are exactly what a RAG system prompt tells the language model. They keep the answer honest, traceable, and safe. Being friendly, or whether the pages sit in one envelope or another, does not change whether the librarian sticks to the pages. That is why only the first three rules belong to the grounding set.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
A creative or brainstorming assistant that should blend its own knowledge with sources A strict ground only directive suppresses the model's useful parametric knowledge, hurting tasks where synthesis beyond the documents is the point.
A single-document summarization task where the whole source is already in context Citation and refusal directives add overhead with little benefit when there is exactly one source and no retrieval step that can miss.
Latency-critical chat where every output token costs budget Per-claim chunk citations inflate output length and cost; you may drop cite chunks and keep only ground only and refusal.
Quick scan · 2–4 min
Everything important, quickly.
3 min: name the three grounding directives, explain the failure each prevents, map them to faithfulness/attribution/abstention, and dismiss the distractors as orthogonal concerns.
Code example
text
You are a question answering assistant. Use ONLY the information in the
numbered context chunks below to answer. Do not use prior knowledge.
RULES:
1. Answer strictly from the provided context.
2. After each claim, cite its source as [chunk_number], e.g. [2].
3. If the context does not contain enough information to answer,
reply: "I don't have enough information to answer that."
CONTEXT:
[1] {chunk_1}
[2] {chunk_2}
[3] {chunk_3}
QUESTION: {user_query}
Side-by-side
Directive
Failure it prevents
Metric it moves
Ground-only
Blending training data facts into the answer
Faithfulness
Cite-chunks
Unverifiable, unattributable claims
Attribution / auditability
Refuse-if-insufficient
Confident answers from a retrieval miss
Abstention / trust
Friendly tone (distractor)
Nothing about grounding
UX, not faithfulness
Real-world examples
Real products, models, and research that use this idea.
1Perplexity 2026 answers cite numbered sources inline and decline cleanly when its retrieval surfaces no supporting pages.
2Anthropic's Claude with the Citations API emits chunk-level source references, the cite chunks directive enforced at the API level.
3Glean's enterprise assistant grounds replies in workspace docs and abstains rather than guessing when the index lacks a relevant document.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you verify the model actually obeys the cite chunks directive instead of fabricating citation tags?
Sign in or go Pro to see the approach for this follow-up.
QRefusal directives often make the model over-refuse on answerable queries. How do you tune that?
Sign in or go Pro to see the approach for this follow-up.
QIf ground only and the model's training knowledge conflict, which should win and how do you enforce it?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Selecting the tone or message placement options. Those are UX or architecture choices; they do not change whether the model grounds its answer in the retrieved context.