What makes video understanding harder than single-image understanding for a VLM?
A VLM that handles single images is being extended to video. Explain why video is fundamentally harder, and what design choices you have to make to keep it feasible.
Every video frame costs as many tokens as a whole image, so full resolution every frame blows the context window — and unlike a still, video also forces you to model temporal order and motion.
Imagine describing a photo versus describing a whole movie to a friend over the phone. The photo is one scene you can detail. The movie is thousands of scenes, and you cannot read every single one aloud — you would run out of breath and time. So you pick a handful of moments and describe those, maybe in less detail, to keep it manageable. But there is a second twist: the movie also has an order. Things happen, then other things happen because of them. With a photo there is no before or after, but with a movie your friend needs the moments in sequence to follow the story. Video is harder on both counts — too much to say, and an order you must preserve.
Detailed answer & concept explanation~6 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
Spend about 5 minutes: lead with the per-frame cost blowup and the two budget levers, then give equal time to temporal order and motion as the second, distinct challenge.
Real products, models, and research that use this idea.
- Gemini long-video understanding — samples frames across long clips rather than encoding every frame, balancing coverage against the token budget.
- Video-LLaVA — encodes a fixed set of sampled frames and aligns them so the LLM can reason over the sequence, not isolated images.
- Keyframe-based pipelines — use scene-change detection to spend the frame budget where content actually shifts instead of sampling uniformly.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhen would uniform sampling fail badly, and how does keyframe sampling fix it?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Treating video as 'just a batch of images' and forgetting temporal modeling. Without frame order and motion cues, the model describes isolated stills and misses actions and cause and effect.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.