How does an agent access a RAG vector store without modifying its own weights?
The agent calls a retrieval tool, the runtime runs the similarity search, and the matching chunks come back as an Observation injected into context. No weights change.
Imagine you are taking an open book exam. You do not memorise the textbook by rewiring your brain. Instead, when a question comes up, you flip to the right page, read it, and use what you just read to answer. After the exam, your brain is exactly the same as before. A RAG vector store is that textbook for an agent. The agent does not retrain itself to learn new facts. When it needs to remember something, it asks the store for the most relevant pages, and those pages get pasted into what it is currently reading. The model then answers using that freshly pasted text. Once the task is over, the model is unchanged. The knowledge lived in the book, not in the model.
Detailed answer & concept explanation~8 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
Start by naming the tool-call path and the Observation injection, then explain non-parametric memory and why weights stay frozen. Cover what RAG memory cannot do: no true update, no real forgetting, and recall bounded by retrieval quality. Close with the production metric you track to catch silent misses.
| Property | RAG vector memory | Fine-tuning into weights |
|---|---|---|
| Update cost | Cheap insert into the store | Full or adapter training run |
| True overwrite | No, stale and new chunks coexist | Yes, weights are replaced |
| Forgetting | Manual deletion from the index | Hard, knowledge is entangled |
| Recall limit | Bounded by retrieval quality | Bounded by model capacity |
| Provenance | Per-chunk source id available | None, knowledge is opaque |
Real products, models, and research that use this idea.
- Claude Opus 4.7 agents in the Anthropic SDK reach long term memory by calling a retrieval tool whose results return as a tool-result block, exactly the Observation path described here.
- LangGraph agents wire a vector store like Pinecone or pgvector behind a retrieve node, so each loop turn can pull episodic memories into state without any retraining.
- ChatGPT's memory feature stores user facts externally and injects relevant ones into context per conversation, illustrating non-parametric memory at consumer scale.
- Mem0 and Letta, formerly MemGPT, package agent long term memory as a retrieval backed store with explicit write, search, and eviction operations over a frozen LLM.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you implement updating a fact when the store has no true overwrite?
QRecall is bounded by retrieval quality. How do you measure and defend against silent misses?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Thinking retrieval changes the model's weights or that the store is wired into attention. Retrieval is just a tool call whose result lands in context as an Observation, then vanishes.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.