Walk through what happens to an input image before a language model can reason over it. What are the stages, and why does the LLM end up treating the image as extra tokens rather than pixels?
Free · 2 AI evals / day
A language model can only do one thing: attend over a sequence of vectors. An image is a 2D grid of pixels, so it has to be reshaped into that sequence first.
The model splits the image into fixed-size **patches**, like cutting a photo into a grid of small squares. Each patch becomes a vector, and a vision transformer (usually a CLIP-pretrained ViT) processes all of them so each output vector understands its part of the image in context.
Those vectors live in the encoder's own dimension, which does not match the LLM's. So a connector, a small projection layer, maps them into the LLM's token embedding space. Now each visual vector has the same shape and meaning as a text token.
Finally, the visual tokens are placed next to the text tokens, and the LLM attends over both together. The reason the image becomes tokens rather than pixels is simple: the language model has no machinery for pixels. It only knows how to attend over embedding vectors, so the image is converted into vectors that look, to the model, exactly like word tokens.
The image goes through three transformations before the language model touches it: patchify, encode, project.
First the image is split into a grid of fixed-size patches and each patch is linearly embedded into a vector with positional information. Those vectors pass through a vision transformer, almost always a CLIP-pretrained **ViT**, where self-attention mixes information across patches so each output vector is context-aware rather than describing one isolated square.
Then a connector maps those encoder outputs into the LLM's token embedding space. This step is necessary because the encoder's hidden dimension rarely matches the language model's embedding width. After projection, a visual vector is shaped exactly like a text token embedding.
The answer to why the LLM treats the image as tokens rather than pixels is architectural. A decoder LLM has exactly one input interface: a sequence of embedding vectors it attends over. It has no convolution, no pixel reader, no 2D machinery. So the only way an image can enter is to be turned into vectors that occupy positions in the sequence. The model then attends over text and image tokens jointly and decodes as usual. It never sees pixels; it sees projected feature vectors that behave like extra positions in its context.
Quick reference
TL;DR
An image is patchified, encoded by a ViT, then projected into the LLM's embedding space, so the language model attends over visual feature vectors as extra tokens and never touches raw pixels.
Imagine a translator who only understands a specific alphabet. You hand them a photo, but they cannot read pictures. So a helper chops the photo into small tiles, studies all the tiles together to figure out what is going on, and rewrites each tile as a word in the translator's exact alphabet. Now the translator gets a stack that mixes those picture-words with the real text-words, all in one alphabet, and reads straight through. The translator never sees the original photo. They only see the rewritten words. That is why the model thinks of the image as just more words in its sequence, rather than as colors and shapes it has to look at directly.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Tasks needing exact small-text reading or precise object counts at low resolution A fixed token budget compresses fine detail, so dense documents and exact counts often need high-resolution tiling or a dedicated OCR pipeline instead.
Generating images rather than understanding them This encode and project path only reads images; producing pixels requires a diffusion or autoregressive image decoder, a different architecture entirely.
Latency-critical inference where every image adds hundreds of tokens to prefill Each image expands into many visual tokens, so feeding images blows up sequence length and prefill cost compared to a short text query.
Quick scan · 2–4 min
Everything important, quickly.
State the three transformations, explain that the LLM's single interface is a sequence of embedding vectors, and close on why tokens, not pixels, buys free cross-modal attention at the cost of a fixed visual token budget.
Real-world examples
Real products, models, and research that use this idea.
1LLaVA is the textbook open-source example: CLIP ViT encoder, small MLP projection, then the language model attends over the projected tokens.
2GPT-5.5 and Gemini 3.1 Pro are natively multimodal but still convert images into tokens the language stack attends over jointly with text.
3Qwen3-VL turns images into visual tokens via a ViT encoder and projection, adding dynamic resolution so detailed images yield more tokens.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does treating images as tokens make cross-modal reasoning almost free for the LLM?
Sign in or go Pro to see the approach for this follow-up.
QWhat does the fixed token budget for an image cost in practice?
Sign in or go Pro to see the approach for this follow-up.
QHow does a cross-attention connector change the answer to why images become tokens?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Saying the LLM looks at the image directly. It only ever sees projected feature vectors in its own embedding space, sitting in the sequence like extra word tokens.