How does an agent access a RAG vector store without modifying its own weights?
The agent calls a retrieval tool, the runtime runs the similarity search, and the matching chunks come back as an Observation injected into context. No weights change.
Imagine you are taking an open book exam. You do not memorise the textbook by rewiring your brain. Instead, when a question comes up, you flip to the right page, read it, and use what you just read to answer. After the exam, your brain is exactly the same as before. A RAG vector store is that textbook for an agent. The agent does not retrain itself to learn new facts. When it needs to remember something, it asks the store for the most relevant pages, and those pages get pasted into what it is currently reading. The model then answers using that freshly pasted text. Once the task is over, the model is unchanged. The knowledge lived in the book, not in the model.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
An LLM's parametric knowledge is fixed at training time. The weights encode everything the model learned, and at inference those weights do not move. That raises an obvious question for any agent that needs to remember things across tasks, or recall facts that postdate its training cut off. How does it remember anything new without being retrained?
The dominant answer is to attach an external vector store and treat it as long term memory. The agent does not learn new facts by changing itself. It looks them up. This pattern is often called RAG as memory, and the crucial detail is exactly how the retrieved text gets from the store into the model's reasoning. The answer is the same tool-calling machinery the agent already uses for everything else.
This framing matters in interviews because the wrong mental models are seductive. It is tempting to imagine the store is somehow plugged into the model's internals, or that the model quietly learns each thing it retrieves. Neither happens. Keeping the boundary between the frozen model and the mutable index sharp is the whole point, and it is what makes the rest of the design tractable.
The retrieval path, step by step
The flow has four hops, and none of them touch the model's weights. First, the model emits a retrieval tool call. From the model's point of view this is identical to calling a calculator or a web search. It outputs a retrieve call with a natural-language query argument, chosen by the model based on what it currently needs to know.
Second, the runtime takes over. It embeds the query string into a vector using an embedding model, then runs an approximate nearest neighbour search over the stored vectors in the index. The closest vectors correspond to the most semantically similar stored chunks. Crucially, this search happens in ordinary software outside the transformer. The model is paused, waiting for a tool result, exactly as it would wait for any other tool.
Third, the runtime packages the top matching chunks as a tool result and appends them to the conversation as an Observation. Fourth, on the next loop turn the model sees that Observation in its context window and reasons over the freshly injected text.
The whole exchange is just one iteration of the standard observe, reason, act, observe loop. Retrieval is not a privileged operation. It is a tool whose backend happens to be a similarity search. This is also why writing to memory is symmetric. The agent calls a write tool with the text to store, and the runtime embeds it and inserts it into the same index, ready to be surfaced by a future retrieval. Read and write are both just tools, and the model's parameters stay frozen through both.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Property | RAG vector memory | Fine-tuning into weights |
|---|---|---|
| Update cost | Cheap insert into the store | Full or adapter training run |
| True overwrite | No, stale and new chunks coexist | Yes, weights are replaced |
| Forgetting | Manual deletion from the index | Hard, knowledge is entangled |
| Recall limit | Bounded by retrieval quality | Bounded by model capacity |
| Provenance | Per-chunk source id available | None, knowledge is opaque |
Real products, models, and research that use this idea.
- Claude Opus 4.7 agents in the Anthropic SDK reach long term memory by calling a retrieval tool whose results return as a tool-result block, exactly the Observation path described here.
- LangGraph agents wire a vector store like Pinecone or pgvector behind a retrieve node, so each loop turn can pull episodic memories into state without any retraining.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you implement updating a fact when the store has no true overwrite?
Version memories with timestamps or a validity flag, retrieve the freshest, and run a periodic reconciliation pass that soft-deletes superseded chunks so contradictory entries do not both surface.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Thinking retrieval changes the model's weights or that the store is wired into attention. Retrieval is just a tool call whose result lands in context as an Observation, then vanishes.
60 second bullets to scan on the way to the call.
State that retrieval enters the loop as an Observation from a tool call.
Explain why no weights change during retrieval.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.