Pick benchmarks that probe the capability you ship. DocVQA and ChartQA cover document image reading; pair them with an in-domain golden set drawn from your own production documents to catch distribution gaps.
Imagine choosing a swimming coach for a child learning the breaststroke. You would not judge them by their marathon times, their chess ratings, or how well they ski. You watch them coach breaststroke, and ideally with kids similar to yours. Picking a benchmark works the same way. MMMU is a marathon, POPE is chess, MathVista is skiing: all impressive, none predictive of how well a model reads an invoice. DocVQA and ChartQA actually test breaststroke. And a sample of your real invoices is the practice pool that tells you whether the coach can teach your specific child.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Benchmark selection looks like a one-question multiple choice but is actually a portfolio design problem. The right portfolio mixes public benchmarks for cross-model comparability with a private golden set for production decision making. Picking just one is a recipe for shipping models that look great on a leaderboard and fail on the third invoice a customer uploads.
This deep dive covers what each candidate benchmark actually probes, why an in-domain golden set is non-negotiable, and how to assemble the small set of metrics that gate a release.
What each candidate benchmark actually measures
DocVQA uses scanned documents (forms, letters, tables, invoices) with questions whose answers live inside the document image. It rewards models that can read text in low-resolution scans, navigate tabular layouts, and extract specific values. This is the capability a document AI product ships.
ChartQA uses chart images (bar charts, line charts, pie charts) with questions about specific data values, trends, or comparisons. Document AI products that handle reports and dashboards need this capability.
MMMU spans college-subject questions across art, business, science, medicine, and engineering. It rewards general academic reasoning over images. A model can score 75 percent on MMMU and still misread invoice line items, because the failure modes are different.
POPE is a narrow probe for one failure mode: object hallucination in natural photos. Does the model claim a cat exists when shown an empty kitchen? Useful for moderation and captioning products. Document hallucination (inventing line items) is a different failure driven by language-prior overreach, not by object misidentification.
MathVista rewards visual math reasoning: read a geometry diagram, solve the problem. Invoice arithmetic is trivial once values are extracted; the hard part is extraction.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- DocVQA and ChartQA are referenced in Anthropic, Google, and OpenAI model cards as the relevant benchmarks for document understanding on Claude Opus 4.7, Gemini 3.1 Pro, and GPT-5.5
- Document AI vendors like Mindee, Rossum, and Docugami publish per-field accuracy on internal golden sets rather than public leaderboard scores
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you build the golden set for a new document type with no labeled history?
Start small (50 to 100 documents) and label by the team or via a vendor (Scale, Snorkel). Stratify by template family, language, and scan quality so the set covers the production distribution. Re-label samples where annotators disagree, and grow the set as production traffic exposes new edge cases.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Picking the most famous benchmark on the leaderboard regardless of what it actually measures, then being surprised when the highest-scoring model performs poorly on your own document mix.
60 second bullets to scan on the way to the call.
What DocVQA, ChartQA, MMMU, POPE, and MathVista each measure
Why benchmark selection is capability mapping, not picking the hardest leaderboard
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.