A VLM must answer questions over a 200-page scanned report, or over a 30-minute video. Explain why this is harder than a single image, what actually breaks first, and the main strategies for making it feasible along with what each one gives up.
Free · 2 AI evals / day
A single image is manageable, but a 200-page report or a 30-minute video is hard for one reason: **vision tokens accumulate**. Every page or sampled frame costs hundreds to thousands of tokens, and those multiply across the whole document or video. The total blows past the context window long before the same amount of plain text would, and it drives cost and latency up roughly with the token count.
What breaks first is usually the context budget — there simply is not room. Even when it fits, attention quality and recall degrade over a very long span, so the model can lose the detail that matters in the middle.
There are four main escape hatches, and each gives something up. Sampling ingests fewer pages or frames (uniform, or keyframe and page selection), but risks skipping the one unit that holds the answer. Compression shrinks the tokens per page or frame with pooling or a resampler, but loses fine detail. Retrieval embeds pages or frames and pulls only the relevant ones into the model, but depends on retrieval quality and can miss reasoning that spans units. A longer-context backbone buys headroom, but still pays a steep cost and does not make tokens free.
In practice you combine retrieval plus compression, sizing each lever to the task.
A single image is bounded; a 200-page report or a 30-minute video is not, because **vision tokens accumulate per unit**. Each page or sampled frame costs hundreds to thousands of tokens, and the total scales with document length or video duration. That total blows past the context window well before the equivalent text would, and it pushes cost and latency up roughly with token count.
The first thing that breaks is the context budget — the input simply does not fit. Even within budget, attention quality and recall degrade across a huge span, so a detail buried in the middle can be effectively lost.
Four strategies make it feasible, each with a real tradeoff.
**Sampling** ingests fewer pages or frames via uniform sampling or keyframe and page selection. The risk is missing the single unit that holds the answer.
Per-unit compression pools or resamples the vision tokens for each page or frame down to a smaller set. The cost is lost fine detail — small text, dense charts.
Retrieval embeds pages and frames, indexes them, and pulls only the query-relevant ones into the VLM. This depends on retrieval quality. It also tends to miss reasoning that spans multiple units.
A **longer-context backbone** with a bigger window buys headroom. It still pays the steep, roughly quadratic attention cost and does not make tokens free.
None is a silver bullet. The practical answer combines retrieval to narrow what enters the model with per-unit compression to shrink what remains. You size each lever to the task rather than feeding everything at full resolution.
Quick reference
TL;DR
Vision tokens accumulate per page or frame, so long multimodal inputs blow the context window first; the escape hatches — sample, compress, retrieve, or enlarge the backbone — each trade away coverage, detail, or cost.
Imagine you have to summarize a 200-page photo album, but you can only hold a small stack of photos in your hands at once. A single photo was easy. Two hundred won't fit in your hands, so you have to choose how to cope.
You could grab every tenth photo and skim — fast, but you might skip the one photo that matters. You could shrink each photo to a thumbnail so more fit in your hands — but tiny details blur out. You could first flip through and pull only the few photos that relate to the question — great, unless the answer is spread across photos you didn't pull. Or you could get bigger hands — helps, but huge hands are heavy and slow, and you still can't hold infinite photos. A vision model facing a long document or video faces exactly these four choices, and every one of them gives something up.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Short inputs — a few pages or a one-minute clip that fit the window comfortably Sampling, retrieval, and compression add machinery and failure modes for no benefit when the whole input already fits at full resolution.
Needle-in-haystack questions answered with uniform sampling Uniform sampling can skip the single answer-bearing page or frame; for these you need retrieval or dense sampling, not a coarse uniform pass.
Fine-detail tasks (small text, dense tables) under aggressive per-unit compression Pooling or resampling each page to few tokens destroys exactly the small text and structure the task reads, so compression trades away the answer.
Quick scan · 2–4 min
Everything important, quickly.
Lead with the root cause: vision tokens accumulate per page or frame, so total tokens scale with length or duration and exhaust the window before equivalent text would. Distinguish the hard failure (context budget) from the softer ones (quadratic attention cost, lost-in-the-middle recall). Walk the four levers as a tradeoff table — sampling gives up coverage, compression gives up detail, retrieval gives up cross-unit reasoning, a bigger backbone gives up cost. Close by stacking retrieval plus compression and sizing each lever to the question type, since none is a silver bullet.
Real-world examples
Real products, models, and research that use this idea.
1Gemini's long-context window is marketed for hour-long video, but per-frame vision tokens still cap how densely you can sample
2GPT-5.5 and Claude Opus 4.7 vision document Q&A on long PDFs typically retrieve relevant pages rather than feeding all 200 at once
3Q-Former-style resamplers compress an image's patch tokens to a fixed small set — the per-unit compression lever applied at scale
4Video VLMs sample keyframes (uniform or scene-change) instead of every frame, the sampling lever trading coverage for feasibility
5Multimodal RAG over scanned reports embeds page images and pulls the top-k pages into the VLM for the question at hand
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you choose between uniform sampling and retrieval for a given question over a long video?
Sign in or go Pro to see the approach for this follow-up.
QWhy doesn't a 2-million-token context window make the compression and retrieval levers obsolete?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Assuming a long-context model 'just handles' a 200-page document, ignoring that vision tokens accumulate per page and exhaust the window long before the equivalent text would.