How to put a number on object hallucination instead of eyeballing it

Attempt it

Your VLM passes general VQA evals but stakeholders complain it 'makes things up' in image descriptions. Describe how you would measure object hallucination specifically, so you can track it as a real metric rather than an anecdote.

Free · 2 AI evals / day

Object hallucination is when a vision model **describes objects that are not actually in the image**. A photo of an empty kitchen counter gets described as having a coffee mug and a phone that simply are not there. The complaint that a model 'makes things up' usually means exactly this. The key idea is that you cannot measure it by reading outputs and trusting your gut. You need images with ground-truth labels — a list of which objects really are present. Then you compare what the model claims against that list and count the false positives. There are two common ways to do this. The first scores free-form captions: you let the model describe the image, pull out the object words it used, and check what fraction of them are not in the ground-truth list. The second turns it into yes or no questions: you ask 'is there a dog?' for objects that are present and for objects that are absent, then score how often the model wrongly says yes. The second style matters because it deliberately includes plausible but absent objects, which is exactly where a model tends to guess yes. Run either method on a labeled set, report a hallucination rate, and re-run it on every model change so it becomes a tracked metric.

Object hallucination is a specific failure: the model confidently asserts objects that are not present in the image. General VQA accuracy hides it because a model can answer many questions correctly on average while still inventing extra objects in its descriptions. To track it as a real metric, you compare claimed objects against ground-truth annotations and isolate the false positives. Two standard styles dominate. **CHAIR-style** metrics work on free-form captions. You generate a description, extract the object nouns, and map them to a fixed object vocabulary. Then you compute the fraction of mentioned objects that are not in the image's ground-truth set. That gives a direct false-positive rate over described objects, reported per-instance and per-sentence. **POPE-style** evaluation reframes hallucination as a balanced discrimination task. Instead of free generation, you pose yes/no questions — 'Is there a {object} in the image?' — half about present objects and half about absent ones. The absent objects are sampled adversarially: random, popular (frequently-occurring objects), or co-occurring (objects that usually appear alongside the present ones). You then score precision, recall, F1, and the yes-ratio, which exposes a model that is biased toward answering yes. In practice you fix a labeled benchmark and report the hallucination rate as a single tracked number. You then wire it into CI so any regression on a new checkpoint or prompt is caught before it ships.

Quick reference

TL;DR

Object hallucination is measured by comparing the objects a model claims against ground-truth labels — CHAIR scores false objects in free captions, POPE probes present/absent with yes-no questions.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a teacher checking a kid's drawing of a park against a photo of the same park. The kid says it shows a dog, a bench, and a kite. The teacher looks at the photo: bench, yes; dog, yes; kite, nope, there was no kite. So one of three things was made up. That is how you grade a vision model for making things up. You take pictures where you already know exactly what is in them, then check whether the model adds objects that were never there. You can let it describe freely and count the invented bits, or you can flat-out ask 'is there a kite?' and see if it wrongly says yes. Counting those wrong yeses gives you a real number instead of a vague feeling.

Key concepts

Multimodal AI LLM Evaluation

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

Open by separating object hallucination from general factual error, then explain why average VQA accuracy hides it. Lay out the two measurement styles: CHAIR over free captions and POPE over balanced present/absent yes-no questions. Spend time on POPE's adversarial negative sampling, since the co-occurring absent objects are where the language prior overrides the pixels. Close on operationalizing it as a tracked regression gate on a fixed labeled set, and note that decoding settings move the number.

Real-world examples

Real products, models, and research that use this idea.

GPT-5.5 and Gemini 3.1 Pro vision teams report hallucination rates separately from VQA accuracy because strong VQA scores still hide over-description
POPE's adversarial negative sampling (co-occurring absent objects) is the standard stress split for VLM hallucination papers
CHAIR was built on COCO's fixed 80-object vocabulary, which is why the metric depends on closed-vocabulary caption parsing
Claude vision and open VLMs like LLaVA are commonly compared on POPE F1 and yes-ratio in 2026 leaderboards
Document-AI teams track hallucinated entities (invented line items on an invoice) as the same false-positive metric applied to text fields

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does adversarial negative sampling in POPE expose more hallucination than random sampling?

Sign in or go Pro to see the approach for this follow-up.

QHow would you measure hallucination of attributes and relations, not just object presence?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Trusting overall VQA accuracy as a hallucination metric, when a model can score well on average while still confidently inventing extra objects in every description.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.