A VLM that handles single images is being extended to video. Explain why video is fundamentally harder, and what design choices you have to make to keep it feasible.
Every video frame costs as many tokens as a whole image, so full resolution every frame blows the context window — and unlike a still, video also forces you to model temporal order and motion.
Imagine describing a photo versus describing a whole movie to a friend over the phone. The photo is one scene you can detail. The movie is thousands of scenes, and you cannot read every single one aloud — you would run out of breath and time. So you pick a handful of moments and describe those, maybe in less detail, to keep it manageable. But there is a second twist: the movie also has an order. Things happen, then other things happen because of them. With a photo there is no before or after, but with a movie your friend needs the moments in sequence to follow the story. Video is harder on both counts — too much to say, and an order you must preserve.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
On paper, extending an image VLM to video sounds like a small step: you already encode images, and a video is just more images. That intuition is exactly what makes the question a good filter. The naive plan — encode every frame and let the LLM sort it out — collides with two hard realities, one about cost and one about meaning.
The cost reality is that video multiplies the single most expensive thing in a VLM, the vision token, by the frame count. The meaning reality is that a video is not a set of images; it is an ordered process, and most of what you want to know about a video lives in the changes between frames, which no single frame contains.
This deep dive takes each in turn. First, why per-frame token cost forces sampling and resolution choices. Then, why temporal order and motion are a genuinely new modeling problem. Finally, how the two pressures combine into a single budget allocation you tune per task — with concrete frame and token numbers so the tradeoff is not abstract.
Why the token budget detonates before anything else
The first wall is arithmetic. A VLM encodes each frame the same way it encodes a standalone image, so each frame costs the same vision-token price — commonly a few hundred to a couple thousand tokens depending on resolution.
Now count frames. Video is typically 24 or 30 frames per second, so a single minute is well over a thousand frames, and a 10-minute clip is tens of thousands. Multiply tens of thousands of frames by hundreds of tokens each and you are at millions of vision tokens. No context window holds that, and even if one did, processing every token would make prefill latency and KV cache cost absurd.
This is why 'feed the whole video at full resolution' is not a real option — it is the canonical wrong answer. The cost does not grow gently with clip length; it grows by a full image's worth of tokens for every single frame. Recognizing that the per-frame multiplier is the problem is the first thing a strong answer establishes, because it is what forces every design choice that follows.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Gemini long-video understanding — samples frames across long clips rather than encoding every frame, balancing coverage against the token budget.
- Video-LLaVA — encodes a fixed set of sampled frames and aligns them so the LLM can reason over the sequence, not isolated images.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhen would uniform sampling fail badly, and how does keyframe sampling fix it?
Use a long static clip with one brief critical event; uniform sampling may miss it, while scene-change detection concentrates frames where content shifts.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating video as 'just a batch of images' and forgetting temporal modeling. Without frame order and motion cues, the model describes isolated stills and misses actions and cause and effect.
60 second bullets to scan on the way to the call.
Why per-frame token cost makes full resolution every frame infeasible
Frame sampling strategies: uniform versus keyframe or scene-change
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.