Your VLM-based document extractor reads invoice paragraphs and headers accurately, but tables come back with merged columns, values attached to the wrong rows, and occasional invented cells. Diagnose the likely causes and lay out the fixes you would try first.
Tables fail when small fonts get downscaled, when tiling cuts through the grid, and when free-form output lets the language prior invent values.
Imagine reading a price list through a foggy window that was also chopped into puzzle pieces. The paragraphs above and below are easy because they are big and continuous. The price list has tiny numbers and a strict grid where every number has to land in the right row and column. When the window blurs the small numbers, the model squints and makes them up. When the puzzle cut runs right through the middle of the table, the model loses track of which column is which. The fix is to clean the window, send the table as one whole piece, and demand answers in a strict form like row by row, so the model cannot just write whatever sounds right.
Detailed answer & concept explanation~8 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
7 min: three failure modes, fix order, layout detection, structured output, validation gates, evaluation methodology.
Real products, models, and research that use this idea.
- Microsoft Table Transformer is the open-source standard for table detection and structure recognition in 2026.
- Surya layout provides general document layout detection that pairs cleanly with Claude Opus 4.7 or GPT-5.5 for cell extraction.
- Anthropic's Claude Sonnet 4.6 with structured output via tool use gives constrained JSON tables that resist hallucination.
- Reducto and Unstract ship production document extraction pipelines built around the detect-crop-extract-validate pattern.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you handle tables that span multiple pages?
QWhat if the model returns the right cells but in the wrong row order?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Asking the model for markdown tables and trusting the result. Markdown is free-form; the model fills gaps from its language prior, inventing plausible cell values that totals never catch.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.