Select the changes that genuinely reduce object hallucination in a vision-language model
Real fixes for object hallucination strengthen the visual signal, weaken the language prior, or verify claims — raising temperature or dropping the encoder do the opposite.
Imagine a kid describing a photo they can barely see. They guess that a dinner table has a fork, because tables usually do — even when this one is empty. To fix it, you can show them a sharper photo, give them practice with tricky photos where the usual thing is missing, or have a second kid double-check each thing they named. What does not help: telling them to guess more wildly, or taking the photo away and making them describe it from the file's name. Those make the guessing worse. The trick is always: make the eyes count for more than the habit.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Multi-select questions reward a different skill than short answers: you cannot just produce a good answer, you have to evaluate six specific claims and decide which earn a checkmark. The fast way to do that wrong is to judge each option on surface plausibility. "Higher resolution sounds good, verification sounds good, temperature... sure, why not." That intuition-matching gets you a couple of distractors every time.
The reliable way is to derive a filter from the root cause first, then run every option through it. For object hallucination the root cause is sharp: the language prior overpowering weak visual grounding. Once you hold that mechanism, each option stops being a standalone judgment call and becomes a yes/no — does this strengthen grounding, weaken the prior, or verify the result?
We will build that filter, apply it to all six options, and pay special attention to the two traps, because the traps are where the question earns its difficulty. Each fails on the exact axis the question is about, which is what makes them tempting and what makes spotting them the real test.
Derive the filter from the mechanism
Before touching the options, state the mechanism in one sentence: object hallucination is a competition between the visual evidence and the language prior, and hallucination is what happens when the prior wins. A table "should" have a fork, a kitchen "should" have a person, so when the model cannot clearly see the patches, it completes the plausible scene from text-side co-occurrence statistics instead of reporting what is there.
That one sentence is a complete filter. Any genuine fix has to do one of exactly three things: make the visual evidence stronger so it wins more often, make the prior weaker so it loses more often, or check the output after the fact so a wrong win gets caught. Call them strengthen, suppress, verify. Hold those three words and you no longer need intuition about any individual option — you need only ask which bucket it falls into, and whether it moves that bucket in the right direction.
Notice what the filter rejects automatically. Anything that adds randomness does not touch either signal — it just spreads the same imbalance over more samples, so it lands in none of the three buckets. Anything that removes visual evidence makes the prior win by default, which is the opposite of strengthen. You can now classify all six options without re-deriving anything.
This is the general move for mechanism-grounded multi-selects, and it is worth internalizing beyond this one question. The slow, error-prone path judges each option on its own surface plausibility, where a well-written distractor will always sound reasonable. The fast, reliable path spends the first minute deriving a filter from the root cause, then runs every option through it mechanically. The filter does the discriminating for you, and a distractor that was engineered to sound plausible has nowhere to hide once you ask which of strengthen, suppress, or verify it actually performs.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- VLM eval suites like POPE probe object hallucination by asking yes/no on present and absent objects — the hard-negative case the grounding-data fix targets.
- Visual contrastive decoding methods reduce hallucination in LLaVA-style models by contrasting image-conditioned logits against a distorted-image baseline.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow does contrastive decoding actually down-weight the language prior at generation time?
Describe contrasting two next-token distributions — one conditioned on the real image, one on a blank or distorted image — and subtracting the latter's logits so tokens driven by the prior rather than the image are penalized.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Picking 'raise the temperature' as a fix — it adds randomness, not faithfulness, and tends to increase hallucination on a model already too willing to invent.
60 second bullets to scan on the way to the call.
The core mechanism: language prior overpowering weak visual grounding
Why hard-negative grounding data attacks co-occurrence skew
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.