Reasoning over a 200 page document or a long video: what breaks, and what are the escape hatches?

Short answer·Hard·4.0 · 0·~3 min·Asked atBaseten Deloitte Perplexity

Attempt it

A VLM must answer questions over a 200 page scanned report, or over a 30 minute video. Explain why this is harder than a single image, what actually breaks first, and the main strategies for making it feasible along with what each one gives up.

Free · 2 AI evals / day

A single image is manageable, but a 200-page report or a 30-minute video is hard for one reason: **vision tokens accumulate**. Every page or sampled frame costs hundreds to thousands of tokens, and those multiply across the whole document or video. The total blows past the context window long before the same amount of plain text would, and it drives cost and latency up roughly with the token count. What breaks first is usually the context budget — there simply is not room. Even when it fits, attention quality and recall degrade over a very long span, so the model can lose the detail that matters in the middle. There are four main escape hatches, and each gives something up. Sampling ingests fewer pages or frames (uniform, or keyframe and page selection), but risks skipping the one unit that holds the answer. Compression shrinks the tokens per page or frame with pooling or a resampler, but loses fine detail. Retrieval embeds pages or frames and pulls only the relevant ones into the model, but depends on retrieval quality and can miss reasoning that spans units. A longer-context backbone buys headroom, but still pays a steep cost and does not make tokens free. In practice you combine retrieval plus compression, sizing each lever to the task.

A single image is bounded; a 200-page report or a 30-minute video is not, because **vision tokens accumulate per unit**. Each page or sampled frame costs hundreds to thousands of tokens, and the total scales with document length or video duration. That total blows past the context window well before the equivalent text would, and it pushes cost and latency up roughly with token count. The first thing that breaks is the context budget — the input simply does not fit. Even within budget, attention quality and recall degrade across a huge span, so a detail buried in the middle can be effectively lost. Four strategies make it feasible, each with a real tradeoff. **Sampling** ingests fewer pages or frames via uniform sampling or keyframe and page selection. The risk is missing the single unit that holds the answer. Per-unit compression pools or resamples the vision tokens for each page or frame down to a smaller set. The cost is lost fine detail — small text, dense charts. Retrieval embeds pages and frames, indexes them, and pulls only the query-relevant ones into the VLM. This depends on retrieval quality. It also tends to miss reasoning that spans multiple units. A **longer-context backbone** with a bigger window buys headroom. It still pays the steep, roughly quadratic attention cost and does not make tokens free. None is a silver bullet. The practical answer combines retrieval to narrow what enters the model with per-unit compression to shrink what remains. You size each lever to the task rather than feeding everything at full resolution.

TL;DR

Vision tokens accumulate per page or frame, so long multimodal inputs blow the context window first; the escape hatches — sample, compress, retrieve, or enlarge the backbone — each trade away coverage, detail, or cost.

Memory aid

Easy to grasp

Imagine you have to summarize a 200-page photo album, but you can only hold a small stack of photos in your hands at once. A single photo was easy. Two hundred won't fit in your hands, so you have to choose how to cope. You could grab every tenth photo and skim — fast, but you might skip the one photo that matters. You could shrink each photo to a thumbnail so more fit in your hands — but tiny details blur out. You could first flip through and pull only the few photos that relate to the question — great, unless the answer is spread across photos you didn't pull. Or you could get bigger hands — helps, but huge hands are heavy and slow, and you still can't hold infinite photos. A vision model facing a long document or video faces exactly these four choices, and every one of them gives something up.

Key concepts

Multimodal AI LLM System Design

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Ask a candidate about long-context multimodal and the weak answer is 'use a model with a bigger context window.' That answer mistakes the symptom for the disease. The reason a 200-page scan or a 30-minute video is hard is not that windows are small — it is that vision is expensive per unit, and a long input multiplies that expense by hundreds or thousands of units.

This deep dive builds the reasoning from the token economics up. We will establish why vision tokens accumulate and why that is qualitatively different from long text, separate the hard failure (the window) from the softer ones (cost and recall), then walk through the four escape hatches as an explicit tradeoff table — what each one buys and what each one gives up — and finish on how to stack them and size each lever to the actual question.

Why vision tokens accumulate, and why text doesn't

Start from the unit cost. A single image, after the vision encoder tiles it into patches, becomes hundreds to thousands of tokens. That is fine for one image. The problem is that a document or a video is not one image — it is a sequence of them.

A 200-page scanned report is 200 images. A 30-minute video sampled at one frame per second is 1,800 images. Each carries its own block of vision tokens, and they add up linearly with the number of units:

T_\text{total} \approx N_\text{units} \times t_\text{per-unit}

The contrast with text is the whole point. The text on a page might be 500 tokens, but the image of that page can be 1,000 to 2,000 vision tokens — and a scanned report has no extractable text at all, so you are stuck paying the image price. Long text context is a known, well-tooled problem; long multimodal context is harder because the per-unit cost is multiples higher and the unit count is large. The token budget you thought was generous evaporates against a few hundred pages.

What breaks first, and what breaks quietly after

Levers one and two: sample and compress

Levers three and four: retrieve and enlarge

Stacking the levers and sizing them to the question

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Gemini's long-context window is marketed for hour-long video, but per-frame vision tokens still cap how densely you can sample
GPT-5.5 and Claude Opus 4.7 vision document Q&A on long PDFs typically retrieve relevant pages rather than feeding all 200 at once

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you choose between uniform sampling and retrieval for a given question over a long video?

Route on question type. Holistic questions — 'summarize the video,' 'what is the overall mood' — favor uniform sampling plus compression because the answer is distributed and any dense slice represents the whole. Needle questions — 'when does the speaker mention the merger' — favor retrieval, because the answer lives in a few frames and uniform sampling can skip them. A hybrid samples coarsely for context and retrieves finely for the specific moment.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Assuming a long-context model 'just handles' a 200-page document, ignoring that vision tokens accumulate per page and exhaust the window long before the equivalent text would.

60 second bullets to scan on the way to the call.

State the root cause: vision tokens accumulate per page or frame, scaling with length or duration
Explain why this exhausts the context window before the equivalent text would

Primary sources. Browse if you want the original framing.