Documents full of charts and tables: how do you keep that information retrievable?
Your source documents (financial reports, datasheets, scientific PDFs) carry critical information in figures, charts, and tables. A plain text extraction pipeline loses it. Explain why, and describe two approaches to make that content retrievable.
Text-only parsing drops figures and tables so they never enter the index; either caption/serialize them into text to embed, or embed the page images with a multimodal model and read them with a vision LLM.
Imagine photocopying a magazine but the copier only captures words, not pictures or charts. Anything shown in a graph just vanishes from your copy, so later you can never find it. To fix it you have two choices. One, have someone look at each chart and write a caption underneath: 'sales doubled from 2023 to 2024,' so the words are now searchable. Two, keep the actual picture and use a helper who can read pictures, so when you search you get the right image back and the helper reads it for you. RAG over chart-heavy documents works the same way: turn pictures into words, or keep the pictures and use a model that can see.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Some of the most valuable information in enterprise documents never appears as prose. A bank's filing puts its growth story in a chart; a chip datasheet puts its specs in a table; a research PDF puts its result in a figure. A RAG system that quietly drops all of that will confidently answer 'I don't have that information' while the answer sits in plain view on the page.
This deep dive starts at the parse step to show exactly how figure and table content is lost, then builds up the two repair strategies — turning visuals into text, and embedding the images themselves — and ends with the 2026 reality that the strongest systems route per modality rather than picking one approach for everything.
Where the data dies: the parse step
The failure happens before retrieval ever runs, which is the single most important thing to establish. A text-extraction pipeline is built to pull a reading-order stream of characters out of a document. It is good at prose and blind to everything else.
A chart is the clearest case. A bar chart encoding 'revenue doubled from 2023 to 2024' contains that fact only as pixels — bar heights and axis labels rendered as graphics. A text extractor sees no characters there, so it emits nothing. The fact is simply absent from the output.
A table fails more insidiously, because it looks like it survived. The extractor pulls the cell text, but the two-dimensional structure collapses. Headers detach from their columns, multi-row cells interleave, and numbers stream out in an order that no longer maps to rows. '$4.2B' might now sit next to the wrong year. The tokens exist but the meaning is scrambled.
Either way, the index never receives a faithful representation. And this is the crux: retrieval can only return what was indexed. A better embedding model, a cross-encoder reranker, a larger top-k — none of them can surface content that was never stored. The fix has to move upstream to ingestion.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- ColPali embeds rendered PDF page images with late interaction, retrieving layout-heavy pages without brittle text parsing.
- Financial-report RAG commonly serializes tables to markdown for precise numeric lookup while keeping page images for charts.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhen does table serialization to text beat embedding the page image, and vice versa?
Tables with precise numeric answers favor serialization to markdown, because text retrieval and the generator can read exact cell values and it supports keyword lookup. Charts, dense layouts, and figures where the visual relationship matters favor image embedding, because captioning would discard detail the vision model can read directly from the page.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Assuming a better retriever or reranker can recover chart and table data. If the parser never put that content in the index, no retriever can surface what was never stored.
60 second bullets to scan on the way to the call.
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.