How does an image become a sequence of tokens an LLM can attend over?

A language model can only do one thing: attend over a sequence of vectors. An image is a 2D grid of pixels, so it has to be reshaped into that sequence first. The model splits the image into fixed-size **patches**, like cutting a photo into a grid of small squares. Each patch becomes a vector, and a vision transformer (usually a CLIP-pretrained ViT) processes all of them so each output vector understands its part of the image in context. Those vectors live in the encoder's own dimension, which does not match the LLM's. So a connector, a small projection layer, maps them into the LLM's token embedding space. Now each visual vector has the same shape and meaning as a text token. Finally, the visual tokens are placed next to the text tokens, and the LLM attends over both together. The reason the image becomes tokens rather than pixels is simple: the language model has no machinery for pixels. It only knows how to attend over embedding vectors, so the image is converted into vectors that look, to the model, exactly like word tokens.

The image goes through three transformations before the language model touches it: patchify, encode, project. First the image is split into a grid of fixed-size patches and each patch is linearly embedded into a vector with positional information. Those vectors pass through a vision transformer, almost always a CLIP-pretrained **ViT**, where self-attention mixes information across patches so each output vector is context-aware rather than describing one isolated square. Then a connector maps those encoder outputs into the LLM's token embedding space. This step is necessary because the encoder's hidden dimension rarely matches the language model's embedding width. After projection, a visual vector is shaped exactly like a text token embedding. The answer to why the LLM treats the image as tokens rather than pixels is architectural. A decoder LLM has exactly one input interface: a sequence of embedding vectors it attends over. It has no convolution, no pixel reader, no 2D machinery. So the only way an image can enter is to be turned into vectors that occupy positions in the sequence. The model then attends over text and image tokens jointly and decodes as usual. It never sees pixels; it sees projected feature vectors that behave like extra positions in its context.