Match each input modality to the front end that turns it into model ready embeddings

Match pairs·Easy·4.0 · 0·~2 min·Asked atCanva N8n Xai

Attempt it

Drag each answer to line up with its matching prompt

Image

Whisper style encoder over log mel spectrogram frames

Audio

ViT style patch encoder, often CLIP or SigLIP pretrained

Video

Subword tokenizer feeding the LLM's embedding table

Text

Sampled frames pushed through a vision encoder, with temporal pooling or position tags

A modern multimodal model is not a single network that takes in pixels and waveforms directly. Each modality has its own front end that converts raw input into vectors the language model can consume. **Images** go through a Vision Transformer (ViT) style encoder. The image is cut into fixed-size patches, each patch is linearly projected into a vector, and a transformer processes the sequence. CLIP and SigLIP are widely used pretrained ViT-style encoders because they were trained to align with text and produce semantically useful features. **Audio** is almost always converted to a log-mel spectrogram first. This is a two-dimensional representation where one axis is time and the other is frequency. Whisper popularized the pattern of using a transformer encoder over spectrogram frames, and most modern audio front ends follow that lineage. **Video** reuses the image encoder. The video is sampled into a handful of frames, and each frame goes through the vision encoder. The resulting sequences are pooled or tagged with temporal positions so the language model knows the order. **Text** is the simplest: a subword tokenizer like BPE or SentencePiece splits the string and the resulting tokens index into the language model's embedding table directly.

Multimodal architectures factor cleanly into two layers: per-modality front ends that turn raw inputs into vectors, and a shared language model that operates on those vectors. - **Image** uses a **ViT-style patch encoder**. Patchify the image, linearly project each patch, run a transformer. CLIP and SigLIP are the dominant pretrained options because their contrastive training already produced text-aligned representations. The output is a sequence of patch tokens. - **Audio** uses a **spectrogram-based encoder**. The waveform is converted to a log-mel spectrogram. A transformer encoder in the Whisper lineage then processes it. Whisper turns the spectrogram into a sequence of frame embeddings with a small 1D convolution stem. Spectrogram-patch encoders such as AST are the ViT-like variant. The output is a sequence of audio frame tokens. - **Video** reuses the image encoder over **sampled frames**, often one to four frames per second, with temporal pooling or explicit time-position tags. Native video models like Gemini 3.1 Pro use richer temporal modelling, but the basic pattern of frame sampling plus vision encoder dominates open-source implementations. - **Text** uses a **subword tokenizer** (BPE, SentencePiece, tiktoken) feeding directly into the LLM's embedding table. There is no separate encoder. The tokens index into the same embedding matrix the model already has. A small projection or connector module sits between each non-text front end and the language model. It maps the encoder's output dimension into the language model's hidden size. Q-Former in BLIP-2 and MLP projectors in the LLaVA family are common implementations.

TL;DR

Each modality gets its own front end: ViT or CLIP for images, Whisper-style spectrogram encoder for audio, sampled frames plus temporal info for video, subword tokenizer for text.

Memory aid

Easy to grasp

Think of a multilingual office where every visitor speaks a different language. Everyone has to write their question on the same kind of form before the manager reads it. Pictures go to a clerk who cuts them into tiles and translates each tile. Sound goes to a clerk who first draws a picture of the sound (a spectrogram) and then translates that. Videos go to the picture clerk, one snapshot at a time, with a stamp saying when each snapshot was taken. Text is only retyped into the standard form. Once everything is on the same kind of form, the manager (the language model) reads it all together and answers.

Key concepts

Multimodal AI Embeddings

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Modern multimodal models look like a single system from outside: you paste an image, an audio clip, a video, and some text into a prompt, and the model reads them all together. Inside, the architecture is more modular than the surface API suggests. Each non-text modality has its own dedicated front end that translates raw input into a sequence of vectors. Only after that translation does the language model take over.

Knowing which front end goes with which modality is the basic mental model for multimodal cost, quality, and architecture choice. It also explains why some modalities are cheaper to add than others, why pretrained image encoders dominate, and why video is mostly an image problem in production today.

Image: ViT-style patch encoders and the CLIP family

The standard image front end is a Vision Transformer. The image is cut into a regular grid of patches, typically 14 by 14 or 16 by 16 pixels each. Each patch is linearly projected into a vector, position embeddings are added, and a transformer encoder processes the sequence. The output is a sequence of patch tokens, one per patch (plus an optional class token).

In practice almost nobody trains a vision encoder from scratch. Pretrained options dominate because they came with massive image-text contrastive training that produced text-aligned features for free. CLIP from OpenAI was the original, contrastively trained on hundreds of millions of image-caption pairs. SigLIP from Google improved on it with a sigmoid contrastive loss that scales better to large batches and produces sharper features for downstream multimodal models. Both are open-weights and serve as the front end in many production VLMs.

The encoder output is a sequence of patch token vectors. A small projector (often just an MLP) maps from the encoder's output dimension to the language model's hidden size. The result is appended to the prompt's token stream.

Audio: spectrograms and the Whisper lineage

Video: frame sampling and temporal context

Text: the asymmetric modality

Production realities and model lineup

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Claude Opus 4.7 processes images as 28x28-pixel patches; Anthropic does not publish the encoder design
LLaVA-NeXT pairs a CLIP ViT-L/14 vision encoder with an MLP projector into a Llama-family or Qwen-family LLM, a pattern many open-source builds copy

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat does the projector or connector module actually do, and why is it necessary?

The encoder's output dimension does not generally match the LLM's hidden size. Also, the encoder's feature space is not aligned with the LLM's token embedding space. The projector is a small trainable module (a linear layer, an MLP, or a Q-Former) that maps from one space to the other. It is the interface contract between two pretrained models.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Assuming the language model directly consumes raw images or audio. In fact, a modality-specific front end pre-encodes each modality into vectors, and only the vectors enter the language model's attention stack.

60 second bullets to scan on the way to the call.

Why each modality needs its own front end before the LLM can consume it
What a ViT-style patch encoder produces from an image

Primary sources. Browse if you want the original framing.