- 1Embed the incoming user question with the same multimodal encoder
- 2Parse source PDFs into text chunks plus rendered page images for figures and charts
- 3Rerank the merged candidates and drop near duplicate pages
- 4Generate the answer with citations pointing back to pages and figures
- 5Retrieve top candidates from both the text and image indexes
- 6Assemble an interleaved prompt mixing text chunks with the retrieved page images
- 7Embed text chunks and page images into a shared multimodal embedding space
Parse, embed (corpus), embed (query), retrieve, rerank, assemble, generate. The first two are ingestion; the rest run per query.
Imagine building a library catalog before the library opens. You unpack every book, photograph each page, and write a card for each section. You file cards for text and cards for pictures using the same color-coded system so a searcher can ask in either modality. Once the library opens, a visitor walks up and asks a question. You write a card for their question in the same color code, search through your catalog, pull out the most promising cards, lay the actual pages on a table in front of the visitor, and read them an answer that points back to the page numbers. The first half of that process (unpacking, filing) is ingestion. The second half (asking, searching, answering) runs every time a question arrives.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Multimodal RAG over figure-heavy PDFs is one of the highest-value applications for vision-capable language models, and one of the easiest to build badly. The pipeline has more moving parts than text RAG, and getting the order wrong, mixing ingestion and query work, skipping rerank, retrieving before embedding the question, breaks the system in subtle ways that show up as bad demos rather than crashes.
This answer walks through the seven stages, why they sit in the order they do, and where the boundaries between them are load-bearing. The same factoring shows up in production systems at AWS, Google Cloud, Anthropic reference architectures, and the major open-source RAG frameworks; once you see the pattern, every multimodal RAG codebase looks like an instance of it.
The ingestion-vs-query split
The first two stages run once per corpus and produce the artifacts the rest of the pipeline reads. The remaining five run per user query at interactive latency. The boundary is the vector store: ingestion writes into it, the query path reads from it.
This split matters because the cost structure is asymmetric. Ingestion is dominated by embedding cost, which scales with corpus size and is amortized across every future query. The query path is dominated by retrieval, rerank, and generation costs, which all run per user request and have to fit within a latency budget. Moving work from the query path to ingestion almost always pays off; moving work in the other direction is almost always a mistake.
A common anti-pattern is treating ingestion as ad-hoc one-off scripts and the query path as the real codebase. In production, ingestion is just as much engineering: it has its own schema migrations, its own retries, its own observability, and its own deployments because the embedding model and the parser do change over time. Designing ingestion as a first-class pipeline is what makes corpus refresh and model migration manageable later.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- LlamaIndex and LangChain both ship multimodal RAG templates that follow this exact seven-stage flow, with parsing and embedding in ingest scripts and the rest in a per-query function
- ColPali popularized using image embeddings of full PDF pages instead of text chunks, but the overall stage order stays the same
What an interviewer would ask next. Try answering before peeking at the approach.
QWhere would you insert caching, and what would you cache?
Cache embeddings keyed by content hash, cache retrieval results keyed by query embedding (for popular queries), and cache generation outputs keyed by the assembled prompt hash (for repeat queries). Invalidate on any change to the embedding model, retrieval parameters, or prompt template.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Skipping the per-query embedding step and trying to retrieve directly from raw question text, or running rerank before retrieval, both of which break the data dependency between stages.
60 second bullets to scan on the way to the call.
Why parsing must come before embedding
What multimodal embedding produces and why both modalities go to the same vector store
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.