The right answer is B. The defining difference between a **RAG chatbot** and a vanilla LLM chatbot is the retrieval step that happens before generation.
When a user asks a vanilla chatbot a question, the query goes straight to the LLM. The model answers from whatever it learned during pretraining, which is frozen, possibly out of date, and contains nothing about your private documents.
When a user asks a RAG chatbot, the system first turns the query into an embedding, searches a vector index of pre-embedded document chunks, and fetches the top-k most similar chunks. Those chunks are then passed to the LLM in the prompt along with the query. The model still produces the answer, but now it has fresh, relevant, private context to base it on. It can also cite which chunk each claim came from.
The model itself is the same in both cases. Nothing got fine-tuned or retrained. The only change is what shows up in the prompt at query time.
The defining difference is the **retrieval step** that a RAG chatbot inserts between the user query and the LLM call. A vanilla chatbot routes the query directly to the model and gets an answer drawn from the model's parametric memory. That memory is whatever was baked into the weights during pretraining: frozen, possibly stale, and containing zero private information.
A RAG chatbot adds one operation in between. The query is embedded and an approximate nearest neighbor search runs against a vector index of pre-embedded document chunks. The top-k most relevant chunks are fetched and inserted into the prompt as context alongside the query. The same LLM, unchanged and unfine tuned, now answers from a prompt that contains fresh, relevant, private material.
This is what option C gets wrong, and it is the most common confusion in interviews. RAG does not fine-tune anything. Fine-tuning modifies model weights and is a separate mechanism with its own tradeoffs. RAG never touches the weights. It is purely a prompt-time augmentation that hands the model evidence to ground its answer in.
The consequence is that RAG handles recency, privacy, and citation, all without any change to the underlying model. Recency means new docs are indexed, not retrained. Privacy means your corpus stays in your index. Citation means the chunk numbers are right there in the prompt.
Quick reference
TL;DR
A RAG chatbot inserts a retrieval step before generation, feeding the model relevant chunks from an external index; a vanilla chatbot answers from frozen pretraining weights alone.
A vanilla chatbot is like a knowledgeable friend who answers from memory. Everything they say comes from what they have already read, and that reading stopped at some point in the past. A RAG chatbot is the same friend, but you also hand them a folder of documents before they answer. They flip to the relevant pages, read what is there, and base their reply on what those pages say. The friend (the model) is the same person in both cases. The difference is the folder. That folder is fresh, can include private documents the friend never saw before, and lets them point at the exact page they used. Fine-tuning would be sending the friend back to school to memorize the documents. RAG just hands them the folder when they need it.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Question can be answered by general world knowledge alone A vanilla chatbot with strong pretraining handles general Q&A directly; adding retrieval just slows things down without improving the answer.
Latency budget is sub-200ms end to end The retrieval step adds embedding, vector search, and prompt-assembly latency; very tight budgets sometimes cannot afford the round trip.
Corpus is small enough to live in the system prompt Skip retrieval and put the whole corpus in the prompt; you save infrastructure and avoid retrieval misses, at the cost of higher per-call tokens.
Quick scan · 2–4 min
Everything important, quickly.
3 min: name the retrieval step as the defining difference, contrast it with both fine-tuning and pure parametric Q&A, walk through what the step buys you (freshness, privacy, citations), and call out that the model itself is unchanged.
Side-by-side
Property
Vanilla chatbot
RAG chatbot
Knowledge source
Frozen pretraining weights
External vector index plus pretraining
Freshness
Capped at training cutoff
Updates by re-embedding new chunks
Private documents
Not accessible
Indexed in your own infrastructure
Citations
Cannot attribute per claim
Per-chunk numeric citations
Model weights
Unchanged
Unchanged (same model)
Extra query-time step
None
Embed query, ANN search, fetch chunks
Real-world examples
Real products, models, and research that use this idea.
1ChatGPT with file search using OpenAI's Responses API to retrieve from uploaded documents before generating, while base ChatGPT answers from training data alone.
2Claude.ai with Projects retrieving from a user-uploaded knowledge base via embedding search, contrasted with base Claude using parametric memory only.
3Glean enterprise search inserting retrieved workspace chunks into the prompt for an LLM call, where a plain Claude or GPT chat would have no access to those docs.
4Notion AI and Perplexity both running RAG over their respective indexes; a vanilla Claude or GPT API call answers without that retrieval layer.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhen would you choose fine-tuning over RAG, or use both together?
Sign in or go Pro to see the approach for this follow-up.
QWhat new failure modes does adding a retrieval step introduce that a vanilla chatbot does not have?
Sign in or go Pro to see the approach for this follow-up.
QHow would you decide between RAG and a 1M-token context window for a private documents use case?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Confusing RAG with fine-tuning. RAG does not touch the model's weights; it inserts retrieved chunks into the prompt at query time. Fine-tuning changes the weights themselves and is a different mechanism entirely.