How to put a number on object hallucination instead of eyeballing it

Object hallucination is when a vision model **describes objects that are not actually in the image**. A photo of an empty kitchen counter gets described as having a coffee mug and a phone that simply are not there. The complaint that a model 'makes things up' usually means exactly this. The key idea is that you cannot measure it by reading outputs and trusting your gut. You need images with ground-truth labels — a list of which objects really are present. Then you compare what the model claims against that list and count the false positives. There are two common ways to do this. The first scores free-form captions: you let the model describe the image, pull out the object words it used, and check what fraction of them are not in the ground-truth list. The second turns it into yes or no questions: you ask 'is there a dog?' for objects that are present and for objects that are absent, then score how often the model wrongly says yes. The second style matters because it deliberately includes plausible but absent objects, which is exactly where a model tends to guess yes. Run either method on a labeled set, report a hallucination rate, and re-run it on every model change so it becomes a tracked metric.

Object hallucination is a specific failure: the model confidently asserts objects that are not present in the image. General VQA accuracy hides it because a model can answer many questions correctly on average while still inventing extra objects in its descriptions. To track it as a real metric, you compare claimed objects against ground-truth annotations and isolate the false positives. Two standard styles dominate. **CHAIR-style** metrics work on free-form captions. You generate a description, extract the object nouns, and map them to a fixed object vocabulary. Then you compute the fraction of mentioned objects that are not in the image's ground-truth set. That gives a direct false-positive rate over described objects, reported per-instance and per-sentence. **POPE-style** evaluation reframes hallucination as a balanced discrimination task. Instead of free generation, you pose yes/no questions — 'Is there a {object} in the image?' — half about present objects and half about absent ones. The absent objects are sampled adversarially: random, popular (frequently-occurring objects), or co-occurring (objects that usually appear alongside the present ones). You then score precision, recall, F1, and the yes-ratio, which exposes a model that is biased toward answering yes. In practice you fix a labeled benchmark and report the hallucination rate as a single tracked number. You then wire it into CI so any regression on a new checkpoint or prompt is caught before it ships.