Why a VLM confidently names objects that aren't in the image, and how to cut it down

Picture an empty dinner table. The model says there is a fork. There is no fork — so where did it come from? It came from the **language prior**. The LLM half of the model has read enormous amounts of text and learned that tables usually have forks, kitchens usually have people. When the model cannot see the image clearly, it falls back on that expectation and completes the scene it thinks should be there, instead of reporting what is actually in the pixels. Why is the image unclear? Vision encoders compress an image into a limited budget of features. Low resolution and a small token budget mean fine detail — or the absence of an object — gets lost. Training data skewed toward common scenes makes it worse. To reduce it, attack the imbalance: feed sharper images (higher resolution, tiling), train on hard examples where the expected object is absent, use decoding that down-weights the language guess, and add a step that re-checks each named object against the image.

The root cause is a **language prior** overriding weak visual grounding. The LLM backbone carries strong text-side expectations about what scenes contain — a table usually has a fork, a kitchen usually has a person. When the visual signal is coarse or under-weighted, the model completes the plausible scene from those priors instead of reporting what the patches actually show. Three things make the visual signal weak. A limited vision-token budget compresses the image into too few features, and low resolution loses fine detail and small objects. On top of that, training data skewed toward common co-occurrences teaches the model that certain objects almost always appear together. The model learns to predict the likely scene rather than the actual one. Mitigations attack the imbalance from three sides. Strengthen the signal with higher input resolution or tiling so fine detail survives. Rebalance the data with diverse grounding examples, including hard negatives where an expected object is genuinely absent. Down-weight the prior at decode with contrastive or grounding-aware decoding. And add an explicit grounding or verification step that re-checks each claimed object against the image. In production you also surface uncertainty and let a verifier or human catch confident misses, rather than trusting the first description as truth.