Reasoning over a 200-page document or a long video: what breaks, and what are the escape hatches?

Short answer·Hard·4.0 · 0·~3 min

Attempt it

A VLM must answer questions over a 200-page scanned report, or over a 30-minute video. Explain why this is harder than a single image, what actually breaks first, and the main strategies for making it feasible along with what each one gives up.

Free · 2 AI evals / day

A single image is manageable, but a 200-page report or a 30-minute video is hard for one reason: **vision tokens accumulate**. Every page or sampled frame costs hundreds to thousands of tokens, and those multiply across the whole document or video. The total blows past the context window long before the same amount of plain text would, and it drives cost and latency up roughly with the token count. What breaks first is usually the context budget — there simply is not room. Even when it fits, attention quality and recall degrade over a very long span, so the model can lose the detail that matters in the middle. There are four main escape hatches, and each gives something up. Sampling ingests fewer pages or frames (uniform, or keyframe and page selection), but risks skipping the one unit that holds the answer. Compression shrinks the tokens per page or frame with pooling or a resampler, but loses fine detail. Retrieval embeds pages or frames and pulls only the relevant ones into the model, but depends on retrieval quality and can miss reasoning that spans units. A longer-context backbone buys headroom, but still pays a steep cost and does not make tokens free. In practice you combine retrieval plus compression, sizing each lever to the task.

A single image is bounded; a 200-page report or a 30-minute video is not, because **vision tokens accumulate per unit**. Each page or sampled frame costs hundreds to thousands of tokens, and the total scales with document length or video duration. That total blows past the context window well before the equivalent text would, and it pushes cost and latency up roughly with token count. The first thing that breaks is the context budget — the input simply does not fit. Even within budget, attention quality and recall degrade across a huge span, so a detail buried in the middle can be effectively lost. Four strategies make it feasible, each with a real tradeoff. **Sampling** ingests fewer pages or frames via uniform sampling or keyframe and page selection. The risk is missing the single unit that holds the answer. Per-unit compression pools or resamples the vision tokens for each page or frame down to a smaller set. The cost is lost fine detail — small text, dense charts. Retrieval embeds pages and frames, indexes them, and pulls only the query-relevant ones into the VLM. This depends on retrieval quality. It also tends to miss reasoning that spans multiple units. A **longer-context backbone** with a bigger window buys headroom. It still pays the steep, roughly quadratic attention cost and does not make tokens free. None is a silver bullet. The practical answer combines retrieval to narrow what enters the model with per-unit compression to shrink what remains. You size each lever to the task rather than feeding everything at full resolution.

Quick reference

TL;DR

Vision tokens accumulate per page or frame, so long multimodal inputs blow the context window first; the escape hatches — sample, compress, retrieve, or enlarge the backbone — each trade away coverage, detail, or cost.

Memory aid

Easy to grasp

Imagine you have to summarize a 200-page photo album, but you can only hold a small stack of photos in your hands at once. A single photo was easy. Two hundred won't fit in your hands, so you have to choose how to cope. You could grab every tenth photo and skim — fast, but you might skip the one photo that matters. You could shrink each photo to a thumbnail so more fit in your hands — but tiny details blur out. You could first flip through and pull only the few photos that relate to the question — great, unless the answer is spread across photos you didn't pull. Or you could get bigger hands — helps, but huge hands are heavy and slow, and you still can't hold infinite photos. A vision model facing a long document or video faces exactly these four choices, and every one of them gives something up.

Key concepts

Multimodal AI LLM System Design

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

Lead with the root cause: vision tokens accumulate per page or frame, so total tokens scale with length or duration and exhaust the window before equivalent text would. Distinguish the hard failure (context budget) from the softer ones (quadratic attention cost, lost-in-the-middle recall). Walk the four levers as a tradeoff table — sampling gives up coverage, compression gives up detail, retrieval gives up cross-unit reasoning, a bigger backbone gives up cost. Close by stacking retrieval plus compression and sizing each lever to the question type, since none is a silver bullet.

Real-world examples

Real products, models, and research that use this idea.

Gemini's long-context window is marketed for hour-long video, but per-frame vision tokens still cap how densely you can sample
GPT-5.5 and Claude Opus 4.7 vision document Q&A on long PDFs typically retrieve relevant pages rather than feeding all 200 at once
Q-Former-style resamplers compress an image's patch tokens to a fixed small set — the per-unit compression lever applied at scale
Video VLMs sample keyframes (uniform or scene-change) instead of every frame, the sampling lever trading coverage for feasibility
Multimodal RAG over scanned reports embeds page images and pulls the top-k pages into the VLM for the question at hand

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you choose between uniform sampling and retrieval for a given question over a long video?

QWhy doesn't a 2-million-token context window make the compression and retrieval levers obsolete?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Assuming a long-context model 'just handles' a 200-page document, ignoring that vision tokens accumulate per page and exhaust the window long before the equivalent text would.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.