What makes video understanding harder than single-image understanding for a VLM?

Attempt it

A VLM that handles single images is being extended to video. Explain why video is fundamentally harder, and what design choices you have to make to keep it feasible.

Free · 2 AI evals / day

Quick reference

TL;DR

Every video frame costs as many tokens as a whole image, so full resolution every frame blows the context window — and unlike a still, video also forces you to model temporal order and motion.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine describing a photo versus describing a whole movie to a friend over the phone. The photo is one scene you can detail. The movie is thousands of scenes, and you cannot read every single one aloud — you would run out of breath and time. So you pick a handful of moments and describe those, maybe in less detail, to keep it manageable. But there is a second twist: the movie also has an order. Things happen, then other things happen because of them. With a photo there is no before or after, but with a movie your friend needs the moments in sequence to follow the story. Video is harder on both counts — too much to say, and an order you must preserve.

Key concepts

Multimodal AI Inference Optimization Attention

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

Spend about 5 minutes: lead with the per-frame cost blowup and the two budget levers, then give equal time to temporal order and motion as the second, distinct challenge.

Real-world examples

Real products, models, and research that use this idea.

Gemini long-video understanding — samples frames across long clips rather than encoding every frame, balancing coverage against the token budget.
Video-LLaVA — encodes a fixed set of sampled frames and aligns them so the LLM can reason over the sequence, not isolated images.
Keyframe-based pipelines — use scene-change detection to spend the frame budget where content actually shifts instead of sampling uniformly.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhen would uniform sampling fail badly, and how does keyframe sampling fix it?

Sign in or go Pro to see the approach for this follow-up.

QHow does the model actually know the temporal order of the sampled frames once they are in the LLM context?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating video as 'just a batch of images' and forgetting temporal modeling. Without frame order and motion cues, the model describes isolated stills and misses actions and cause and effect.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.