Match each multimodal benchmark to the capability it actually measures

Multimodal benchmarks are not interchangeable. Each one targets a **different skill**, so matching them correctly means knowing what kind of image and what kind of question each one uses. VQA uses natural photos and asks concrete questions like "what color is the bus?" — it tests grounded question answering. DocVQA swaps the photo for a scanned document and asks you to read dense text and layout. ChartQA hands the model a chart or plot and asks it to pull out values and reason over them. These three differ mainly by the type of visual artifact. MMMU and MMBench test breadth, not a single artifact. MMMU is college-level reasoning across many academic subjects, mixing diagrams, formulas, and figures. MMBench checks many fine-grained capabilities one at a time, so you see a profile rather than one number. POPE is the odd one out. It is not an accuracy score in the usual sense — it asks present/absent yes-no questions to catch object hallucination, meaning the model claiming an object is there when it is not. Sorting these correctly comes down to artifact type for the first three, breadth for the middle two, and faithfulness for the last.

The matching turns on **what each benchmark isolates**. Three of them vary by visual artifact, two test breadth, and one tests faithfulness. VQA pairs a natural image with a direct question, so it measures grounded question answering on everyday scenes. DocVQA keeps the question-answering frame but uses scanned documents, so it stresses reading text, tables, and layout. ChartQA again keeps the frame but uses charts and plots, so it stresses value extraction and numeric reasoning over a figure. The throughline for these three: same task shape, different input that the model must parse. MMMU and MMBench are about coverage. MMMU is a hard, college-level set spanning many academic subjects, so a high score signals broad multimodal reasoning rather than one narrow skill. MMBench breaks competence into many fine-grained dimensions and reports per-capability results, which is why teams use it to find specific weak spots. POPE sits apart because it does not reward general accuracy — it probes object **hallucination** with present/absent yes-no questions. A model can score well on VQA and still fail POPE by confidently asserting objects that are not in the image. The reason to keep a hallucination benchmark separate is that aggregate accuracy averages this failure away, so it stays invisible until you measure it directly.