Multiple users complain that your AI assistant 'ignores the documents I uploaded and just makes things up'. Before involving an ML engineer, which pipeline layer should you investigate first?
The complaint is 'the AI ignored my documents and made things up'. The instinct is to blame the search part, but that's usually wrong here.
A RAG system has two separable jobs. First, retrieval finds the right document chunks and places them in the prompt. Second, the model reads those chunks and writes an answer. The user's words point at the second job, not the first.
The layer to check first is the **augmentation prompt**, the instruction that wraps the retrieved chunks before the model sees them. If that prompt never says 'answer only from the context below, cite the chunk you used, and refuse if it's not enough', the model treats the chunks as optional hints. It blends them with what it learned in training, and when the chunks feel thin it guesses.
The giveaway is that the chunks are usually right there in the prompt; the model just wasn't told to stick to them. That makes the fix a small prompt edit, far cheaper than swapping the embedding model or retraining anything. Check the prompt, confirm the chunks were retrieved, then escalate only if grounding directives are already present.
Investigate the **augmentation prompt** first. The complaint 'ignored my docs and made things up' is the classic signature of missing grounding directives, not broken retrieval.
Split the pipeline into two failure classes. Retrieval-missing means the relevant chunks never made it into the prompt; you'd see this as low context recall, and the fix lives in embeddings, chunking, or the vector index. LLM-ignoring means the chunks ARE in the prompt but the model didn't anchor to them; the fix lives in the prompt.
The one diagnostic that separates them: log the actual prompt sent to the model. If the right chunks are present and the model still hallucinated, retrieval is fine and the augmentation layer is the culprit.
A grounding correct prompt does three things. It instructs the model to answer only from the supplied context and to cite which chunk each claim came from. It also tells the model to refuse or say 'I don't know' when the context is insufficient. Without these, the model defaults to parametric memory whenever the context is weak. The fix is often a three-line system-prompt change, which is why it's the right first stop, before touching the model version or the vector database.
Quick reference
TL;DR
Check the augmentation prompt first. 'Ignores my docs and makes things up' usually means the chunks were retrieved fine but the prompt never told the model to ground in them.
Imagine you hand a student a folder of notes and say 'answer the exam'. If you never tell them 'only use these notes', they'll mix the notes with whatever they already remember, and when the notes feel thin they just guess confidently. That's exactly what an LLM does in a RAG system. The retrieval step often works fine, the right pages got pulled and placed in front of the model. But the instruction wrapping those pages, the augmentation prompt, never said 'answer only from this context, tell me which page you used, and refuse if it isn't enough'. So the model treats the pages as optional hints and falls back on its training. When users say 'it ignored my upload', the cheapest first check is that instruction, not the search engine and not the model.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
The prompt-payload log already shows the correct chunk is absent for failing queries That's a retrieval-recall problem; editing grounding directives can't help when the right chunk never reached the model.
Users report 'it says it can't find anything' rather than confident wrong answers A no-information response points at retrieval or index coverage, not at the model ignoring present context.
The augmentation prompt already carries answer only, cite, and refuse directives and is correctly placed The cheapest lever is exhausted; escalate to model version, context crowding, or chunk-conflict analysis instead.
Quick scan · 2–4 min
Everything important, quickly.
3 min: split retrieval-missing from LLM-ignoring, name the prompt-payload inspection step, list the three grounding directives, then justify the prompt-first triage ordering.
Side-by-side
Signal
Retrieval-missing
LLM-ignoring (this case)
What broke
Relevant chunks never reach the prompt
Chunks reach the prompt but aren't used
Prompt-payload check
Correct chunk absent from payload
Correct chunk present in payload
Owning layer
Embeddings, chunking, vector index
Augmentation prompt grounding directives
User-visible feel
'It couldn't find anything'
Confident answer that ignores the upload
First fix
Tune retrieval / reindex / hybrid search
Add answer only / cite / refuse directives
Real-world examples
Real products, models, and research that use this idea.
1A Claude- or GPT-backed support assistant that hallucinates despite correct retrieval, fixed by adding 'answer only from the provided context and cite the source chunk' to the system prompt.
2Notion AI or Glean answering from general knowledge instead of workspace docs because the grounding instruction was too weak.
3A LangChain or LlamaIndex RAG app where the default prompt template lacked a refusal clause, so the model filled gaps from training memory.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QYou logged the prompt payload and the correct chunk IS present, but the model still fabricates even after you added grounding directives. What now?
Sign in or go Pro to see the approach for this follow-up.
QHow would you build an automated test that catches grounding regressions before they reach users?
Sign in or go Pro to see the approach for this follow-up.
QIf retrieval recall and grounding are both weak, which do you fix first and why?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Jumping straight to the embedding model or vector DB. Those change retrieval recall. This symptom is a grounding failure: the chunks were found, the prompt just never forced the model to use them.