Product wants users to type a phrase like 'the demo where the dashboard crashes' and jump to the exact moment across a 100,000 hour video library. Architect the system: ingestion, indexing, query path, and the cost levers that make it feasible.
Three signal tracks at ingest (shots, ASR, OCR), all keyed by (video, timestamp). Hybrid retrieve, temporally cluster the hits, VLM-rerank the top clips. Sampling rate and precomputation are the cost levers.
Imagine a library with 100,000 hours of video. You cannot watch them all every time someone asks a question. Instead, when each video arrives, you do three things: take snapshots at the moments where the scene changes, write down everything that gets said with timestamps, and copy any text that appears on screen. You file all three by video and time. Later, when someone asks 'where does the dashboard crash', you check what was said, what was on screen, and what the snapshots look like, all at once. The matches cluster around a specific minute. You hand back a clip that starts right there.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Searching for a moment inside 100,000 hours of video is the canonical multimodal retrieval problem at scale. The user types a phrase and expects to land in the right minute of the right video. The library is too large to scan at query time, and a single signal (only audio, only visuals) is too thin to handle the variety of queries users actually ask. The architecture that works is ingest-heavy, query-light, and multi-modal: three parallel signal tracks at ingest, all keyed by moment, retrieved in parallel and fused temporally at query time.
This walkthrough lays out the full system: ingestion pipeline per video, indexing strategy, query path with fusion and rerank, the cost levers that make 100k hours feasible, and the evaluation methodology that lets you tune the system without flying blind.
Mental model: the library is huge, queries are cheap, ingest is paid once. Spend everywhere at ingest; spend nothing avoidable at query time.
Ingestion: three signal tracks, one moment key
Per video, the ingest pipeline produces three parallel tracks that all index against (video_id, time_offset).
Visual track. Run shot detection (TransNetV2 or PySceneDetect ContentDetector). Each shot gets one keyframe; long shots over 30 s get supplemental frames at 1 fps. Embed each keyframe with a multimodal embedding model (SigLIP-family, Voyage Multimodal 3, Cohere Embed v4). Adaptive sampling is the difference between $0.05/hour and $0.50/hour at scale.
Audio track. Run ASR with word-level timestamps (Whisper-large-v3, Deepgram Nova-3, AssemblyAI Universal-2). Chunk the transcript into overlapping windows (20 s windows, 10 s stride). Embed each chunk with a text embedding model. Word-level timestamps let you snap query results to precise moments rather than coarse paragraphs.
On-screen text track. Run OCR (PaddleOCR, Surya OCR, or VLM-based) on every keyframe. Capture slide text, error dialogs, console output, lower-thirds, anything rendered as pixels. Embed OCR text per keyframe.
Why all three. A query like 'the demo where the dashboard crashes' uses all three signals: 'demo' is in spoken context (audio), 'dashboard' is a visual scene (visual), 'crashes' may be on-screen text ('Fatal Error', stack trace) or spoken. Any single track has blind spots.
Storage. Three vector indices (Qdrant, Weaviate, pgvector, hosted Pinecone equivalents). Every entry carries (video_id, t_start, t_end, modality, source_text_or_caption). The payload is small; the embeddings are the bulk.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Google's Video Intelligence API and YouTube search internally combine shot detection, ASR, and OCR for moment-level retrieval.
- Twelve Labs Pegasus and Marengo offer hosted multimodal video search APIs built on the same architecture.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do you handle adding new videos to a large existing index?
Incremental indexing per video, never reindex existing content; route ingest jobs to a separate cluster from query traffic; version the embedding model so a future model upgrade can dual-write before cutover.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Indexing whole videos instead of moments. The deliverable is a jump-link to a timestamp, not a video title; the index key must be (video_id, time_offset) end to end.
60 second bullets to scan on the way to the call.
The three signal tracks (visual, audio, on-screen text) and why each matters
Why shot detection plus adaptive sampling beats uniform high-rate sampling
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.