A video VLM uniformly samples 16 frames from a clip. The vision encoder turns each sampled frame into 256 vision tokens (no high res tiling). Ignoring the text prompt, how many vision tokens does this clip contribute to the LLM's context?
Vision tokens for a sampled clip = frames sampled × tokens per frame. Here 16 × 256 = 4096. The clip's real length and frame rate never enter the math once you fix the sample.
Imagine making a flip-book of a long movie. You do not trace every single film cell — you pick a handful of pages and draw only those. The work you do depends on how many pages you picked and how detailed each drawing is, not on how long the original movie ran. Here the model picks 16 pages from the clip, and each page becomes 256 little stickers it can read. So the total stickers are just 16 times 256, which is 4096. The movie could be one minute or one hour — once you decide to keep 16 pages, the rest of the movie stops mattering for the count.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
This looks like a trivial multiplication, and the arithmetic is — but the reason it is trivial is the whole lesson. Candidates who get this wrong usually do so by reaching for the impressive-sounding number in the problem, the raw frame count of the source clip, instead of the number that actually drives cost.
The point of the question is to test whether you understand what frame sampling buys. Sampling is not a minor preprocessing detail; it is the design decision that makes video tractable for a language model at all. Once you internalize that, the cost formula falls out and the distractor numbers become obviously irrelevant.
This deep dive walks through the per-frame cost model, why sampling decouples cost from clip length, how the two budget levers multiply, and what a real token budget feels like when you do the full calculation with concrete numbers. The arithmetic is a one-liner; the understanding behind it is what an interviewer is actually checking, and it generalizes to every video token-budget estimate you will make on a real system.
The per-frame cost model behind the multiplication
A video VLM does not have a special 'video encoder' that ingests a clip whole. It reuses the same image encoder it uses for single pictures, and applies it frame by frame to a chosen set of frames.
Each frame, once encoded, becomes a fixed number of vision tokens — the same way a single image does. In this scenario that number is 256 tokens per frame. Those tokens are appended to the LLM's context just like text tokens, and the model attends over all of them together.
So the clip's contribution to the context is purely additive across the kept frames. With 16 frames at 256 tokens each, you append 16 blocks of 256 tokens, which is 16 × 256 = 4096 tokens. There is no cross-frame discount in the base model — each kept frame pays its full per-frame price. That is why the operation is a multiplication and never a sum that involves the parts of the video you discarded.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Gemini video understanding — samples frames from long videos at a chosen rate, and the sampled-frame count drives the token budget, not the raw clip length.
- LLaVA-style video models — encode a fixed number of sampled frames per clip, each into a set patch-token count, so cost is frames × tokens.
What an interviewer would ask next. Try answering before peeking at the approach.
QYou only have a 4096-token budget for the whole video. Do you keep 16 frames at 256 tokens, or 32 frames at 128 tokens?
Frame it as temporal coverage versus spatial detail under a fixed product; tie the choice to whether the task is action recognition or fine-grained reading.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Multiplying by the clip's original frame count (e.g. 18,000) instead of the 16 sampled frames. Sampling exists precisely so the source length drops out of the token math.
60 second bullets to scan on the way to the call.
The cost formula: sampled frames times tokens per frame
Why the clip's true length and frame rate drop out of the math
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.