Diagnose why your document extractor nails paragraphs but mangles invoice tables
Your VLM based document extractor reads invoice paragraphs and headers accurately, but tables come back with merged columns, values attached to the wrong rows, and occasional invented cells. Diagnose the likely causes and lay out the fixes you would try first.
Tables fail when small fonts get downscaled, when tiling cuts through the grid, and when free-form output lets the language prior invent values.
Imagine reading a price list through a foggy window that was also chopped into puzzle pieces. The paragraphs above and below are easy because they are big and continuous. The price list has tiny numbers and a strict grid where every number has to land in the right row and column. When the window blurs the small numbers, the model squints and makes them up. When the puzzle cut runs right through the middle of the table, the model loses track of which column is which. The fix is to clean the window, send the table as one whole piece, and demand answers in a strict form like row by row, so the model cannot just write whatever sounds right.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Document extraction with vision language models is one of the strongest use cases for the technology: a single model handles arbitrary layouts, mixed languages, and handwritten annotations that traditional OCR pipelines struggle with. It is also one of the easiest places to fool yourself into thinking the model is working when it is not. Paragraphs flow naturally and forgive small errors. Tables do not. A value attached to the wrong row changes the total; an invented cell flows into accounting; a misread column header propagates through every downstream system.
The pattern of bugs in the question (merged columns, wrong rows, invented cells) is diagnostic. It points at three specific failure modes that compound, and the fix order is the same across vendors, models, and document types. This walkthrough goes through each failure, the leverage available to fix it, and how to validate that the fix actually moved the metric you care about.
Mental model: paragraphs are forgiving. Tables are a 2D grid where every cell must land in the right row and the right column. Anything that blurs the grid (low resolution, bad tile cuts, free-form output) corrupts every downstream value.
Failure mode 1: resolution starvation
Invoice line items are typeset at 8 to 12 px cap height in the original PDF. When you render the PDF to an image at 72 DPI and the VLM provider then downscales further to fit its tile budget, those characters end up 3 to 4 pixels tall. That is below the resolving power of any modern vision encoder.
What the model sees in that region is a blur. What it outputs is whatever its language prior thinks invoices usually say at that position: plausible prices, common SKU patterns, round-number totals. The output looks fine until you reconcile against the actual document.
The fix: render at higher DPI. 200 to 300 DPI rather than 72 to 96 DPI roughly doubles the pixel height of small fonts. Save a debug crop of the table region after preprocessing and verify the small text is legible to your own eye; if you cannot read it, the encoder cannot either.
Don't just crank DPI globally. Token cost scales with pixel area. The right move is to detect table regions and render those at higher DPI while keeping the rest of the page at standard DPI. That selectively spends tokens where they matter.
Why this is the first fix: if the model cannot see the cell, no amount of prompt engineering or structured output will recover the value. Resolution is the foundation.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Microsoft Table Transformer is the open-source standard for table detection and structure recognition in 2026.
- Surya layout provides general document layout detection that pairs cleanly with Claude Opus 4.7 or GPT-5.5 for cell extraction.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you handle tables that span multiple pages?
Layout detection identifies table continuation; merge crops vertically before extraction or run extraction per page and stitch rows with header matching.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Asking the model for markdown tables and trusting the result. Markdown is free-form; the model fills gaps from its language prior, inventing plausible cell values that totals never catch.
60 second bullets to scan on the way to the call.
Why resolution starvation invents plausible cell values
How tile boundaries through tables break column alignment
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.