Your team is comparing a vanilla LLM chatbot against a RAG chatbot. What is the single most important difference between them at query time?
A RAG chatbot inserts a retrieval step before generation, feeding the model relevant chunks from an external index; a vanilla chatbot answers from frozen pretraining weights alone.
A vanilla chatbot is like a knowledgeable friend who answers from memory. Everything they say comes from what they have already read, and that reading stopped at some point in the past. A RAG chatbot is the same friend, but you also hand them a folder of documents before they answer. They flip to the relevant pages, read what is there, and base their reply on what those pages say. The friend (the model) is the same person in both cases. The difference is the folder. That folder is fresh, can include private documents the friend never saw before, and lets them point at the exact page they used. Fine-tuning would be sending the friend back to school to memorize the documents. RAG just hands them the folder when they need it.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The vanilla versus RAG comparison is one of the most commonly mis answered RAG interview questions, and the reason is that the two systems look almost identical from the outside. A user types a question. A model produces an answer. There is no visible difference in the chat UI, no obvious tell in the latency, no separate logo. The difference is one architectural step that you cannot see from the user side, and naming that step precisely is the whole point of the question.
The failure mode this question filters for is candidates who conflate RAG with fine-tuning. This is the option-C trap, and it is more common than it should be. Fine-tuning modifies the model. RAG never touches the model. The two are different mechanisms with different cost curves, different update cadences, different failure modes, and different operational concerns. Mixing them up signals the candidate has read about RAG without building one.
This deep dive walks through what each system actually does at query time, why the retrieval step is the defining feature, what the step enables that a vanilla chatbot fundamentally cannot do, and why the model choice is independent of the pattern.
What each system does at query time
A vanilla LLM chatbot has the simplest possible request path. The user's message is wrapped in whatever system prompt the app uses and sent directly to the model API. The model returns tokens. The app streams them to the user. There is no external store, no lookup, no document context. Everything the model says comes from the parametric memory baked into its weights at training time.
This means the vanilla chatbot's knowledge is frozen at the model's training cutoff. For Claude Opus 4.7 or GPT-5.5 in mid-2026, that cutoff is somewhere in early 2026. It also means the chatbot knows nothing about your private documents, your customers, your internal procedures, or anything that was not in the public training data. It will happily answer questions about those things by guessing or extrapolating, but every such answer is unverifiable by construction.
A RAG chatbot inserts one operation between the user message and the model call. The user query is embedded using a model like text-embedding-3-large or Voyage Embed v3, then used to query a vector index containing pre-embedded chunks of your documents. The top-k most similar chunks are fetched, formatted into a context block, and inserted into the prompt alongside the user query. The model now sees both the question and the relevant evidence.
The model itself is identical. Claude Opus 4.7 in the vanilla path is the same Claude Opus 4.7 in the RAG path. No weights changed. No fine-tuning happened. The only difference is the prompt the model receives, and the only difference in the prompt is the retrieved chunks.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Property | Vanilla chatbot | RAG chatbot |
|---|---|---|
| Knowledge source | Frozen pretraining weights | External vector index plus pretraining |
| Freshness | Capped at training cutoff | Updates by re-embedding new chunks |
| Private documents | Not accessible | Indexed in your own infrastructure |
| Citations | Cannot attribute per claim | Per-chunk numeric citations |
| Model weights | Unchanged | Unchanged (same model) |
| Extra query-time step | None | Embed query, ANN search, fetch chunks |
Real products, models, and research that use this idea.
- ChatGPT with file search using OpenAI's Responses API to retrieve from uploaded documents before generating, while base ChatGPT answers from training data alone.
- Claude.ai with Projects retrieving from a user-uploaded knowledge base via embedding search, contrasted with base Claude using parametric memory only.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhen would you choose fine-tuning over RAG, or use both together?
Fine-tune for style, format, and behavior; RAG for facts, recency, and citations; combine when you need both consistent voice and fresh evidence.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Confusing RAG with fine-tuning. RAG does not touch the model's weights; it inserts retrieved chunks into the prompt at query time. Fine-tuning changes the weights themselves and is a different mechanism entirely.
60 second bullets to scan on the way to the call.
What the retrieval step does and where it sits in the query flow
Why retrieval enables fresh, private, and citable answers
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.