An LLM's own knowledge is frozen in its **weights**, set at training time and unchanged at inference. So how does an agent remember things it learned yesterday, or facts that did not exist during training? It uses an external store and looks things up, the same way you would consult a notebook.
The mechanism is ordinary tool use. The agent issues a retrieval tool call, for example `search_memory(query)`. The runtime takes that query, embeds it into a vector, finds the closest stored vectors in the database, and returns the matching text chunks.
Those chunks then enter the agent loop as an Observation, exactly like the result of a web search or a database query. The model reads them on the next turn and uses them to reason. Nothing about the model itself changes.
This is why it scales. You can add a million memories to the store without touching the model. The correct answer is that the agent issues a retrieval call and the runtime injects the chunks as the next Observation. The wrong options all imply weight changes or special wiring, which is not how retrieval works.
An agent accesses a RAG vector store through the **tool-calling** path, not through any weight change. The model never reads the store directly. It emits a retrieval tool call, and the runtime does the actual work.
The steps are simple. The model outputs a call such as `retrieve(query)`. The runtime embeds the query, runs a nearest neighbour search over the stored vectors, and pulls back the top matching chunks. Those chunks are appended to the conversation as an Observation.
On the next turn the model sees that Observation in its context window and reasons over it. This is identical to how any other tool result re-enters the loop. Retrieval needs no special API.
The key property is that this injection is transient. The chunks live in the context for the current task and then disappear when the context is cleared. The model's parameters are untouched. This is what people mean by non-parametric memory. Knowledge lives in the external index, not in the weights, so you can grow it without retraining.
The distractors all describe weight level mechanisms. Attention integration, query time fine-tuning, and adapter layers would all modify or bypass the model, which is not what RAG does.
Quick reference
TL;DR
The agent calls a retrieval tool, the runtime runs the similarity search, and the matching chunks come back as an Observation injected into context. No weights change.
Imagine you are taking an open book exam. You do not memorise the textbook by rewiring your brain. Instead, when a question comes up, you flip to the right page, read it, and use what you just read to answer. After the exam, your brain is exactly the same as before. A RAG vector store is that textbook for an agent. The agent does not retrain itself to learn new facts. When it needs to remember something, it asks the store for the most relevant pages, and those pages get pasted into what it is currently reading. The model then answers using that freshly pasted text. Once the task is over, the model is unchanged. The knowledge lived in the book, not in the model.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
The knowledge is small, stable, and fits comfortably inside the context window already. A static prompt or simple file is cheaper and avoids retrieval misses entirely.
You need a fact authoritatively corrected so the old version can never resurface. RAG cannot truly overwrite; stale and corrected chunks coexist and either may be retrieved.
The behaviour you want is a new skill or style, not new facts to look up. Skills live in weights or system prompts; retrieval injects text, it does not change how the model reasons.
Latency budgets are extremely tight and every added round trip matters. A retrieval call adds an embedding plus a search plus an extra loop turn before the model can answer.
Quick scan · 2–4 min
Everything important, quickly.
Start by naming the tool-call path and the Observation injection, then explain non-parametric memory and why weights stay frozen. Cover what RAG memory cannot do: no true update, no real forgetting, and recall bounded by retrieval quality. Close with the production metric you track to catch silent misses.
Side-by-side
Property
RAG vector memory
Fine-tuning into weights
Update cost
Cheap insert into the store
Full or adapter training run
True overwrite
No, stale and new chunks coexist
Yes, weights are replaced
Forgetting
Manual deletion from the index
Hard, knowledge is entangled
Recall limit
Bounded by retrieval quality
Bounded by model capacity
Provenance
Per-chunk source id available
None, knowledge is opaque
Real-world examples
Real products, models, and research that use this idea.
1Claude Opus 4.7 agents in the Anthropic SDK reach long term memory by calling a retrieval tool whose results return as a tool-result block, exactly the Observation path described here.
2LangGraph agents wire a vector store like Pinecone or pgvector behind a retrieve node, so each loop turn can pull episodic memories into state without any retraining.
3ChatGPT's memory feature stores user facts externally and injects relevant ones into context per conversation, illustrating non-parametric memory at consumer scale.
4Mem0 and Letta, formerly MemGPT, package agent long term memory as a retrieval backed store with explicit write, search, and eviction operations over a frozen LLM.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you implement updating a fact when the store has no true overwrite?
Sign in or go Pro to see the approach for this follow-up.
QRecall is bounded by retrieval quality. How do you measure and defend against silent misses?
Sign in or go Pro to see the approach for this follow-up.
QWhen does retrieving memory hurt rather than help the agent?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Thinking retrieval changes the model's weights or that the store is wired into attention. Retrieval is just a tool call whose result lands in context as an Observation, then vanishes.