Why images are expensive to process in a VLM, and the core tradeoff that creates

Short answer·Medium·4.0 · 0·~3 min

Attempt it

Explain why feeding images to a vision-language model costs far more than feeding the same query as text, and name the central tradeoff a team has to manage because of it.

Free · 2 AI evals / day

Images are expensive because **one image is not one token**. The vision encoder turns an image into a block of hundreds of vision tokens, and the language model has to process all of them. High resolution makes it worse. To read fine detail, modern VLMs use tiling: they split the image into a grid of tiles, encode each tile separately, and add a downscaled global view. Each tile is its own block of tokens, so a high-resolution image can become thousands of tokens. Those vision tokens go through the model's prefill exactly like text tokens. Prefill cost grows with how many tokens are in the prompt, so a big image dominates both the prompt size and the time before the model starts answering. A typed question might be 30 tokens; a tiled image might be 2,000. The central tradeoff is **resolution**. More resolution and more tiles buy accuracy on small text, dense charts, and fine detail, but they cost more latency and money on every single request. The practical move is to set a token budget per use case: feed low resolution for simple scene photos and reserve high-resolution tiling for documents and charts where the fine detail actually matters.

Feeding an image costs far more than the same query as text because the image expands into many tokens before the model ever reasons over it. The encoder converts an image into a block of vision tokens, typically hundreds for a single base-resolution pass. High-resolution **tiling** multiplies that: the image is split into a grid of tiles, each tile emits its own block of tokens, and a downscaled global view adds one more block. So a single high-resolution image can run to thousands of tokens, while the user's typed question is a few dozen. Those vision tokens are not cheap metadata. They flow through prefill just like text, and attention cost grows with sequence length, so the image dominates both the prompt size and the prefill latency. The image, not the text, usually decides time to first token. That sets up the central tradeoff: **resolution against cost**. More resolution and more tiles buy accuracy on small text, dense charts, and fine detail, because each region gets a higher-fidelity representation. But every added tile is a full block of tokens charged on every request, in both dollars and latency. The senior move is to set an image-token budget per use case rather than always maxing resolution. A simple scene photo needs far fewer tokens than a dense document scan. Matching the tiling level to the content is how teams keep cost in line with the accuracy the task actually requires.

Quick reference

TL;DR

One image becomes hundreds to thousands of vision tokens that go through prefill like text, so the central tradeoff is resolution: more tiles buy fine-detail accuracy but cost latency and money every request.

Memory aid

Easy to grasp

Imagine you pay a reader per page they read. A short typed question is a single page, quick and cheap. But an uploaded photo is not one page. The model slices it into many little pieces and, if you ask for high detail, slices it into even more pieces so it can read the fine print. Suddenly that one photo is hundreds or thousands of pages, far more than your typed question. The reader has to flip through every page before answering, which takes time and costs money. So the big decision is how finely to slice each image. Slice it coarsely and you save pages but might miss small text. Slice it finely and you catch the detail but pay for all those extra pages.

Key concepts

Multimodal AI Inference Optimization Tokenization

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

Explain that an image expands into hundreds to thousands of vision tokens that prefill like text, name the resolution versus cost tradeoff with its prefill and KV-cache consequences, and close on per use case token budgeting.

Real-world examples

Real products, models, and research that use this idea.

OpenAI GPT-5.5 high-detail images are priced as a base block plus per-tile blocks, so a full-page scan costs many times a thumbnail.
LLaVA-NeXT AnyRes tiling expands a high-resolution image into multiple per-tile token blocks plus a global view, the canonical open-source version of this cost.
Qwen3-VL dynamic resolution scales visual tokens with input size, so teams cap resolution per use case to control prefill cost and latency.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does the resolution versus cost relationship scale roughly with the square of resolution?

QBeyond the per-token price, what makes a high-token image expensive on a serving fleet?

QHow would you architect a high-volume VLM feature to control image cost without losing needed accuracy?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Thinking an image costs about the same as a short text prompt. One image routinely costs hundreds to thousands of tokens, dwarfing the user's typed query.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.