How modern VLMs read a high resolution image that a fixed size encoder can't ingest whole
A vision encoder expects a fixed input size (say 336x336), but the user uploads a 4K screenshot full of small text. Explain how dynamic resolution / AnyRes tiling lets the VLM handle it, and what that approach costs.
AnyRes tiling slices a high-res image into native-size tiles plus one downscaled global view, encodes each, and concatenates the features — buying small-text detail at the price of many more vision tokens.
Imagine reading a giant wall map through a small magnifying glass. You can't see the whole map sharply at once, so you slide the glass across it square by square, reading each patch up close. Then you step back and glance at the whole map once to remember where everything sits. That is what a vision model does with a big screenshot. It chops the picture into small squares it can read clearly, looks at each one, and also keeps a tiny shrunk copy of the full image for context. The catch: every square is extra work, so a huge image takes much longer and costs more to process.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Vision encoders are not flexible about input size. A ViT trained at 336 by 336 carves the image into a fixed grid of patches, and that grid is baked into the position handling and the training distribution. Hand it a 4K screenshot and the only naive option is to resize the whole thing down to 336 — at which point a paragraph of UI text becomes a few unreadable smears.
That collision between fixed encoders and high-resolution inputs is where a huge fraction of real VLM failures live in 2026. Document question answering, chart reading, and computer-use agents all hinge on small text and precise layout, exactly the signal that downscaling throws away. Interviewers probe this because it is the gap between a model that demos well on photos and one that survives a real document workload.
This deep dive covers why the fixed grid is the root problem, how AnyRes tiling restores detail, why the global view is not optional, and how the whole scheme turns into a token-budget decision you have to manage rather than ignore.
Why a fixed encoder grid breaks on high-res input
A Vision Transformer splits its input into a grid of patches, say 16 by 16 pixels each, and turns each patch into one embedding. At 336 by 336 that is a 21 by 21 grid, roughly 441 patch embeddings. The number of patches, and the positions the model was trained to understand, are fixed by that input size.
When the input is far larger, you cannot just feed more patches without leaving the resolution the encoder was trained on. So the default pipeline resizes the whole image down to 336 first. Every pixel that mattered — a 12-pixel-tall menu label, a thin chart axis, a price tag — gets averaged into oblivion before the encoder ever runs.
The key realization is that information is destroyed at the resize step, before any clever modeling. No downstream attention or projection can recover text that was blurred away in preprocessing. That is why the fix has to happen at the image level: you have to get the small regions in front of the encoder at a resolution where they are still legible, instead of asking the model to read a smudge.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
# Vision-token cost of AnyRes tiling
tokens_per_pass = 576 # encoder's native patch-token count
grid_rows, grid_cols = 3, 2 # best-fit tile grid for the image
tiles = grid_rows * grid_cols # 6 detailed tiles
passes = tiles + 1 # + 1 downscaled global view
vision_tokens = passes * tokens_per_pass
print(vision_tokens) # 4032 tokens vs 576 for a plain downscale
# ~7x the tokens, latency, and price -> cap the grid to the taskReal products, models, and research that use this idea.
- LLaVA-NeXT (1.6) — popularized AnyRes: split into native-resolution tiles plus a downscaled global view to read high-res images and documents.
- GPT-5.5 vision — high-detail mode tiles a large image and bills more tokens, while low-detail mode uses a single coarse pass for cheaper, gist-level answers.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you cut the vision-token cost of a dense tile grid without losing readability?
Discuss token pooling or compression on tile features, picking a coarser grid for the task, or routing only document-heavy images to high-detail mode.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Forgetting that tiling multiplies vision tokens. A dense tile grid can turn one image into thousands of tokens, so latency and price climb fast — it is not a free quality upgrade.
60 second bullets to scan on the way to the call.
Why a fixed-size encoder destroys small text on a downscaled high-res image
How tiling keeps each region at the encoder's native resolution
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.