When an LLM in a RAG pipeline behaves differently because of the retrieved context, what is actually changing about the model?

Attempt it

Quick reference

TL;DR

Retrieved context changes the input the model conditions on, not its weights. Same fixed θ, different prompt, different output distribution. No parameter update happens at query time.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Think of the model as a sealed calculator. You can't open it up and rewire it, that's what training does, and it's already done. What you can do is type different numbers in. RAG doesn't rewire the calculator; it just types better numbers in by pasting relevant documents into the prompt. The calculator runs the exact same logic it always did, but because the input is now richer, the answer comes out different and more grounded. People sometimes say in-context learning 'teaches' the model on the fly. It doesn't. Nothing inside the box changes. The model is the same function before and after; only what you fed it changed. That distinction matters a lot when an interviewer asks what RAG actually does to the model.

Key concepts

RAG system design Fine-tuning Prompt engineering

Quick scan · 2–4 min

Everything important, quickly.

4 min: fixed-θ function framing, prompt-as-conditioning vs weight editing, ICL research framings, and why RAG and fine-tuning compose.

Side-by-side

Aspect	RAG / in-context learning	Fine-tuning
What changes	The prompt (conditioning input)	The parameters θ
Persistence	Gone when the query ends	Durable across all future queries
Mechanism	Forward pass only, no gradients	Gradient descent on a training set
Math touched	Posterior P(y\|prompt;θ)	θ itself (and thus the prior)
Best for	Fresh, private, query specific facts	Behavior, format, domain reasoning

Real-world examples

Real products, models, and research that use this idea.

Perplexity feeds retrieved web pages into the prompt of a frozen model (GPT or Claude); same weights, grounded answers, zero retraining per query.
Notion AI 2026 answers over your workspace by injecting retrieved docs into the context window of an unchanged base model.
Claude with the Files API and project knowledge conditions on uploaded documents at inference without any fine-tune of the underlying weights.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QIf no weights change, why does adding few-shot examples to the prompt reliably improve task accuracy?

Sign in or go Pro to see the approach for this follow-up.

QSome papers say transformers do 'gradient descent in the forward pass' during ICL. Does that contradict 'no parameter update'?

Sign in or go Pro to see the approach for this follow-up.

QHow would you decide between RAG and fine-tuning for a domain specific assistant?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.