Check the augmentation prompt first. 'Ignores my docs and makes things up' usually means the chunks were retrieved fine but the prompt never told the model to ground in them.
Imagine you hand a student a folder of notes and say 'answer the exam'. If you never tell them 'only use these notes', they'll mix the notes with whatever they already remember, and when the notes feel thin they just guess confidently. That's exactly what an LLM does in a RAG system. The retrieval step often works fine, the right pages got pulled and placed in front of the model. But the instruction wrapping those pages, the augmentation prompt, never said 'answer only from this context, tell me which page you used, and refuse if it isn't enough'. So the model treats the pages as optional hints and falls back on its training. When users say 'it ignored my upload', the cheapest first check is that instruction, not the search engine and not the model.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
When a user says your AI assistant 'ignored my documents and just made things up', it feels like a search bug. The natural reflex is to open the vector database, stare at the embedding model, or suspect the LLM went off the rails. That reflex costs hours, because the words in the complaint are pointing at a different layer than the one most engineers reach for.
This deep dive teaches the triage skill behind the question: how to map a user-visible symptom to the right pipeline layer, why the augmentation prompt is the correct first stop for this particular symptom, and how one inspection step cleanly separates the two failure classes that produce 'it made things up'. The goal is a runbook you can execute under pressure, ordered cheapest fix first, so you don't escalate to an ML engineer for a problem a three-line prompt change solves.
Two failure classes hide behind one symptom
A RAG pipeline does two separable jobs, and 'the model made things up' can come from either. Conflating them is the root mistake.
The first job is retrieval. The system embeds the query, searches the vector index, and places the top chunks into the prompt. When this job fails, the relevant chunk never reaches the model. We call this retrieval-missing, and it lives in the embedding model, the chunking strategy, or the index configuration. It shows up quantitatively as low context recall.
The second job is grounding. The model reads the chunks that did arrive and writes an answer anchored to them. When this job fails, the chunk is sitting right there in the prompt but the model blends it with training memory or overrides it entirely. We call this LLM-ignoring, and it lives in the augmentation prompt.
The phrase 'ignored my docs and made things up' is the linguistic fingerprint of the second class. Retrieval failures tend to surface to users as 'it said it couldn't find anything in my file'. Grounding failures surface as confident answers that contradict or sidestep the uploaded content. That difference in user phrasing is your first triage signal, before you touch any code.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Signal | Retrieval-missing | LLM-ignoring (this case) |
|---|---|---|
| What broke | Relevant chunks never reach the prompt | Chunks reach the prompt but aren't used |
| Prompt-payload check | Correct chunk absent from payload | Correct chunk present in payload |
| Owning layer | Embeddings, chunking, vector index | Augmentation prompt grounding directives |
| User-visible feel | 'It couldn't find anything' | Confident answer that ignores the upload |
| First fix | Tune retrieval / reindex / hybrid search | Add answer only / cite / refuse directives |
Real products, models, and research that use this idea.
- A Claude- or GPT-backed support assistant that hallucinates despite correct retrieval, fixed by adding 'answer only from the provided context and cite the source chunk' to the system prompt.
- Notion AI or Glean answering from general knowledge instead of workspace docs because the grounding instruction was too weak.
What an interviewer would ask next. Try answering before peeking at the approach.
QYou logged the prompt payload and the correct chunk IS present, but the model still fabricates even after you added grounding directives. What now?
Check directive placement (top vs bottom of prompt), context length crowding out instructions, conflicting chunks, and whether the model version genuinely has weaker grounding; consider few-shot grounding examples or a citation enforced output schema.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Jumping straight to the embedding model or vector DB. Those change retrieval recall. This symptom is a grounding failure: the chunks were found, the prompt just never forced the model to use them.
60 second bullets to scan on the way to the call.
The two RAG failure classes: retrieval-missing vs LLM-ignoring
Why 'ignored my docs' maps to grounding, not retrieval
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.