Measuring whether a vision-language model is actually good: what goes on the scorecard

Short answer·Medium·4.0 · 0·~3 min·Asked atAccenture Modal Labs Runway

Attempt it

Your team is choosing a vision-language model for a product that reads charts and documents and answers questions about them. Beyond a captioning score, what do you actually measure to evaluate the model, and how do you choose which benchmarks matter?

Free · 2 AI evals / day

A captioning score tells you the model can describe an image, but your product needs it to read documents and charts and answer questions. Those are different skills, so the scorecard needs several measures. Start with grounded question answering. VQA-style accuracy checks whether the model answers concrete questions about an image rather than guessing from language priors. Then add the measures that match your job directly: document QA for reading dense text and layout, and chart QA for pulling values out of plots. These are the ones that look like your real inputs. Add a couple of broad-competence checks. MMMU tests hard reasoning across many subjects, and MMBench breaks competence into fine-grained skills so you can see a profile rather than one number. These tell you how generally capable the model is. The most important addition is a **hallucination** metric. POPE-style present/absent checks catch the model inventing objects or values that are not there. Aggregate accuracy averages this away, so you have to measure it on purpose. Finally, weight the benchmarks that mirror your real work, and keep a private set of your own examples that the model has never seen.

The right move is to build a scorecard of distinct capabilities and then weight them by **product fit**, not to chase a single leaderboard number. The scorecard has layers. Grounded question answering (VQA-style) confirms the model can answer concrete questions about an image instead of relying on language priors. The product-matching layer — document QA and chart QA — directly tests reading dense text, tables, and figures, which is the actual job here. A broad-competence layer (MMMU for hard cross-subject reasoning, MMBench for a fine-grained capability profile) tells you general strength and where the weak spots are. A faithfulness layer (POPE/CHAIR-style present/absent and caption-grounding checks) catches confident hallucination, which is the failure mode that hurts most when a dashboard quotes a wrong number. The selection principle is the part interviewers want to hear. Weight the benchmarks whose inputs and failure costs mirror your workload. For a chart and document product, **document and chart QA plus hallucination rate dominate**, while a generic caption or natural-image score is nearly irrelevant. Two cautions finish the answer. Public benchmarks can leak into pretraining, inflating scores that do not transfer, and they rarely match your customers' actual documents. So pair them with a held-out internal eval set drawn from real user data. That internal set is what truly predicts production behavior, while the public numbers just shortlist which models are worth testing.

TL;DR

Build a scorecard of distinct skills — grounded QA, document and chart reading, broad reasoning, and hallucination — then weight the ones that mirror your real inputs and failure costs, backed by a held-out internal set.

Memory aid

Easy to grasp

Imagine hiring someone to read your messy paperwork and answer questions about it. You would not pick them just because they write pretty photo captions. You would test the actual job: hand them a real invoice and a real chart and see if they get the numbers right. You would also check whether they make things up when they are unsure, because a confident wrong answer is worse than "I don't know." Choosing a vision-language model works the same way. You run several small tests that each check one skill, you care most about the ones that look like your real work, and you keep a private stack of your own examples that the model has never seen.

Key concepts

LLM Evaluation Multimodal AI

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Choosing a vision-language model is one of those decisions that looks like a lookup and is actually a design problem. The naive version is to open a leaderboard, sort by the headline number, and pick the top row. For a product that reads charts and documents, that path quietly ships the wrong model surprisingly often, because the headline number rarely measures the job you are hiring the model to do.

The interview is really testing whether you can design an evaluation, not recite benchmark names. A strong answer builds a scorecard of distinct capabilities, weights them by what your product actually consumes and what its failures cost, and then defends the choice with evidence the vendor could not have trained on. This deep dive walks through why a single score misleads, how to layer the scorecard, why hallucination earns its own slot, and how leakage and distribution mismatch push the decisive evidence onto a held-out internal set.

Why one number cannot answer the question

The instinct to rank models by a single score comes from how we evaluate text LLMs, where one accuracy figure on a broad benchmark is a reasonable first cut. Multimodal evaluation breaks that habit because the perception pipeline has many failure surfaces that do not correlate.

A model's handling of natural photos depends on coarse scene understanding. Its handling of scanned documents depends on resolving small text and layout. Its handling of charts depends on precise value extraction where one tick mark is the difference between right and wrong. These lean on different parts of the encoder and connector, so a model can be excellent at one and weak at another. A blended score averages those together and hides exactly the weakness you care about.

There is a second reason. Captioning, the most common headline metric, rewards fluent plausible description. A model can write a beautiful caption that mentions an object that is not present, and the captioning score barely penalizes it. For a product that answers questions about real documents, fluency is not the goal — correctness on specific values is. So the first move is to refuse the single number and ask which separate capabilities the product depends on.

Layering the scorecard by capability

Hallucination is the slot teams forget

Leakage, distribution mismatch, and the internal set

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Document-AI teams building invoice and contract readers weight DocVQA-style accuracy and a faithfulness check above natural-image scores
Analytics products that summarize dashboards run ChartQA-style evals to catch value misreads before they reach users

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you build the held-out internal eval set so it actually predicts production behavior?

Sample from real user inputs across the document and chart types you serve, stratify by difficulty and edge cases (low-res scans, multi-column tables, dense plots), and write a clear rubric. Keep it private and rotated so it cannot leak into any vendor's training. Add human-graded faithfulness checks where a wrong number is high cost.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Picking the model at the top of a generic VQA or captioning leaderboard, then discovering in production that it misreads invoice totals and fabricates chart values.

60 second bullets to scan on the way to the call.

List the capability layers a multimodal scorecard should separate
Explain why a captioning or generic VQA score can mislead for a document product

Primary sources. Browse if you want the original framing.