Measuring whether a vision-language model is actually good: what goes on the scorecard

A captioning score tells you the model can describe an image, but your product needs it to read documents and charts and answer questions. Those are different skills, so the scorecard needs several measures. Start with grounded question answering. VQA-style accuracy checks whether the model answers concrete questions about an image rather than guessing from language priors. Then add the measures that match your job directly: document QA for reading dense text and layout, and chart QA for pulling values out of plots. These are the ones that look like your real inputs. Add a couple of broad-competence checks. MMMU tests hard reasoning across many subjects, and MMBench breaks competence into fine-grained skills so you can see a profile rather than one number. These tell you how generally capable the model is. The most important addition is a **hallucination** metric. POPE-style present/absent checks catch the model inventing objects or values that are not there. Aggregate accuracy averages this away, so you have to measure it on purpose. Finally, weight the benchmarks that mirror your real work, and keep a private set of your own examples that the model has never seen.

The right move is to build a scorecard of distinct capabilities and then weight them by **product fit**, not to chase a single leaderboard number. The scorecard has layers. Grounded question answering (VQA-style) confirms the model can answer concrete questions about an image instead of relying on language priors. The product-matching layer — document QA and chart QA — directly tests reading dense text, tables, and figures, which is the actual job here. A broad-competence layer (MMMU for hard cross-subject reasoning, MMBench for a fine-grained capability profile) tells you general strength and where the weak spots are. A faithfulness layer (POPE/CHAIR-style present/absent and caption-grounding checks) catches confident hallucination, which is the failure mode that hurts most when a dashboard quotes a wrong number. The selection principle is the part interviewers want to hear. Weight the benchmarks whose inputs and failure costs mirror your workload. For a chart and document product, **document and chart QA plus hallucination rate dominate**, while a generic caption or natural-image score is nearly irrelevant. Two cautions finish the answer. Public benchmarks can leak into pretraining, inflating scores that do not transfer, and they rarely match your customers' actual documents. So pair them with a held-out internal eval set drawn from real user data. That internal set is what truly predicts production behavior, while the public numbers just shortlist which models are worth testing.