A finance assistant summarizes uploaded reports, and reviewers keep catching it stating wrong numbers from bar charts: values that look plausible but do not match the chart. The model never says it is unsure. Explain why VLMs hallucinate on charts specifically, and what you would change in the pipeline.
Vision encoders downscale charts past the point where axis labels are readable and patch embeddings blur precise bar heights, so the language prior fills gaps with plausible numbers.
Imagine someone with blurry glasses trying to read a thermometer across the room. They know thermometers usually show numbers like 70 or 72, so when they cannot quite see, they confidently guess 71. They are not lying; their eyes are weak and their brain helpfully fills in what is probably there. Vision models do the same thing on charts. The tick labels are tiny, the bar tops are at fuzzy positions, and the language part of the model knows that revenue charts usually have numbers around a certain range, so it produces a confident guess. The fixes all amount to handing the model a clearer thermometer, or asking it to read the dial step by step, or letting it admit it cannot see.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Charts are unusually hostile to vision-language models because they concentrate the encoder's weakest behaviors: tiny labels, precise spatial comparisons, and answers that depend on reading exact numbers. When the visual signal is weak, the LLM's strong prior over plausible numbers produces a confident guess. This is the hallucination pattern reviewers keep catching.
This deep dive covers the perception failures specifically (because that is where the fix lives), the pipeline change that turns chart understanding into a two-pass flow, and the evaluation discipline that keeps the fix from regressing.
Why charts are hard: three quantization losses
Downscaling at the provider edge. Every commercial VLM has an input resolution ceiling. Claude downscales beyond a 1568 px long edge; GPT-5.5 tiles to roughly 1024 to 2048 px; Gemini 3.1 Pro similar. A 3500 px tall report image is downscaled by a factor of two or more before tokenization. Axis labels that were 12 px tall in the original become 6 px tall after downscaling, which is at or below the readability threshold even for trained models.
Patch tokenization in the vision encoder. Vision transformers split images into patches (commonly 14x14 in SigLIP, 32x32 in efficient variants). Each patch becomes a single token in the encoder. A bar top whose true position lies between two y-axis values gets quantized to one patch's embedding; the encoder has no way to express 'between'.
Vision-token budget into the LLM. The encoder produces hundreds to thousands of vision tokens, but only some subset gets passed to the LLM as cross-attention keys. Detail at the chart level competes for token budget with detail at the page level. The result is that precise numerical evidence is structurally lossy.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- DePlot and ChartReader are specialized chart-to-table models that beat general VLMs on extraction-specific benchmarks
- Claude Opus 4.7 and Gemini 3.1 Pro both show measurable improvement on ChartQA when callers crop the chart region before sending
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you build the chart-to-table extraction prompt so the model abstains on unreadable values?
Specify the schema (axis_title, axis_units, series_name, data points with x and y) and add an explicit field per data point like readable: true/false with a note. Provide few-shot examples where the correct behavior is marking a value as unreadable. Train or fine-tune if production scale justifies; otherwise rely on instruction-following in modern frontier models.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating chart hallucination as a generic hallucination problem instead of a vision-encoder failure, then trying to fix it with prompt engineering alone rather than addressing resolution and transcription.
60 second bullets to scan on the way to the call.
Three quantization steps that compress chart evidence in a VLM pipeline
Why the language prior produces confident plausible numbers when visual evidence is weak
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.