How does a Vision Transformer turn an image into a sequence, and what mask runs over it?

Short answer·Medium·4.0 · 0·~3 min·Asked atCoinbase Meesho Typeface

Attempt it

A Vision Transformer (ViT) applies the same attention machinery as a language model, but the inputs are images. Walk through how the per token sequence is built from an image, what attention mask is used, and how the model knows where each patch came from.

Free · 2 AI evals / day

ViT applies the same transformer attention machinery as a language model, but the inputs are images. The trick is converting the image into a sequence of tokens. Start with an image of shape (H, W, 3), say 224 x 224 x 3 RGB. Cut it into a grid of fixed-size patches, typically 16 x 16 pixels each. A 224 x 224 image gives a 14 x 14 grid, which is 196 patches. Flatten each patch into a vector of length 16 x 16 x 3 = 768 numbers, then pass it through one linear projection to get a d_model-dimensional embedding. The sequence of 196 patch embeddings is what attention runs over. For classification, ViT prepends a learnable [CLS] token to the sequence (so the input is 197 tokens total: 196 patches plus 1 CLS). After all transformer layers, the final hidden state of the CLS token gets fed to a classification head. The attention mask is FULL bidirectional. Every patch attends to every other patch in both directions. There is no causal mask because there's no notion of 'future' in a static image, the patches all exist simultaneously and any patch can in principle inform any other. Position is injected via learned 2D positional embeddings added to the patch embeddings. Without PE, ViT would be permutation-invariant over patches, which would destroy the spatial structure the model needs.

**The pipeline.** An image of shape (H, W, 3) is split into a grid of fixed-size patches (typically 16 x 16 pixels). For a 224 x 224 image that's a 14 x 14 grid, 196 patches. Each patch is flattened (16 * 16 * 3 = 768 values for RGB) and passed through ONE linear projection that produces a d_model-dimensional token embedding. The output is a sequence of patch tokens. **The CLS token.** For classification ViT prepends a learnable [CLS] token, making the sequence 197 tokens long. After all the transformer layers run, the final hidden state of the CLS token is the pooled image representation that feeds the classification head. The CLS token is not strictly required (some variants use mean-pooling over patch tokens instead), but it's the convention from the original ViT paper. **The attention mask: FULL bidirectional.** Every patch can attend to every other patch in both directions. There is no causal mask because an image has no temporal direction, patches exist simultaneously and any pair can be structurally related (a corner can inform the center, the center can inform a corner). Applying a causal mask would arbitrarily block half the attention based on the raster order of patches, which would destroy structural reasoning. **Position encoding.** Self-attention is permutation-invariant over input tokens. Without position information, ViT couldn't distinguish patch (0, 0) from patch (10, 10), they'd be the same token. The original ViT adds learned 2D positional embeddings to the patch embeddings before the transformer. Swin and MaxViT use relative position biases inside attention instead. Either way, position has to be injected, otherwise the spatial structure is lost. The rest of the architecture (transformer encoder, LayerNorm, FFN, output projection) is identical to a BERT-style encoder.

TL;DR

Split the image into fixed-size patches, linearly project each into a token, prepend a CLS token, run full bidirectional attention, and add learned 2D positional embeddings.

Memory aid

Easy to grasp

Imagine cutting a photograph into a grid of small square tiles, like a jigsaw puzzle. You hand the model the tiles in a stack along with a sticky note on each one saying 'I came from row 3, column 7'. The model looks at every tile and lets each one see every other tile, no rules about which tiles came first because all tiles exist at once. After all that looking, one special blank tile (the CLS token) collects everything the model figured out about the whole picture and hands the summary off for classification.

Key concepts

Vision Transformer Patch tokens Bidirectional attention Positional encoding

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Vision Transformer (ViT) was the design that demonstrated transformer attention transfers cleanly from language to vision, given enough data. The core trick is small: cut the image into patches, project each patch to a token embedding, then run the same transformer encoder that BERT uses. The architectural simplicity is almost suspicious, the original paper's tagline was 'an image is worth 16 x 16 words', and that's literally the entire bridge.

The interesting questions are in the details. How big should patches be? Where does positional information come from? Why bidirectional attention rather than causal? How does the CLS token work? What happens at inference time when the resolution changes? Each detail has a defensible answer and a few common misconceptions.

This deep dive walks through the patch tokenization pipeline, the choice of bidirectional attention, the position encoding options, hierarchical variants like Swin, and the downstream applications (CLIP, DINOv2, SAM, multimodal LLMs) that built on the ViT foundation.

From image to token sequence

The image to token conversion has three steps: split, flatten, project. Plus the CLS token and position embeddings.

The split

An image of shape (H, W, 3) is partitioned into a grid of non-overlapping patches of size P x P pixels. The number of patches is (H / P) x (W / P). For a 224 x 224 image with P = 16, that's 14 x 14 = 196 patches.

Flatten and project

Each P x P x 3 patch is flattened to a vector of length P^2 * 3 (768 for P = 16), then passed through a single linear projection to produce a d_model-dimensional embedding. In code this is usually implemented as a single Conv2d(3, d_model, kernel_size=P, stride=P) which does the patching and projection in one efficient operation.

Prepend the CLS token

A learnable d_model-dimensional vector is prepended to the patch sequence. This [CLS] token has no input data; it's pure parameter that the model uses to pool global information across all patches. After all transformer layers, the CLS token's final hidden state is the pooled image representation that feeds the classification head.

Alternatives: mean-pooling over patch tokens works comparably; DINOv2 uses both CLS and patch-mean for different downstream tasks; register tokens (Darcet et al. 2024) add a few additional learnable tokens that absorb attention noise.

Add positional embeddings

A learnable position embedding of shape (1, num_patches + 1, d_model) is added to the token embeddings. This is critical: without it, ViT is permutation-invariant over patches.

The pipeline is mechanically simple. The interesting question is why this works at all, given the loss of locality and translation-equivariance that CNNs explicitly bake in.

Why bidirectional attention and not causal

Position encoding: how the model knows where each patch is

Beyond vanilla ViT: hierarchical variants and downstream applications

Situations where this technique stops working.

2–4 min · Everything important, quickly.

python

import torch
import torch.nn as nn

class PatchEmbed(nn.Module):
    def __init__(self, img_size=224, patch_size=16, in_chans=3, d_model=768):
        super().__init__()
        self.proj = nn.Conv2d(in_chans, d_model,
                              kernel_size=patch_size, stride=patch_size)
        num_patches = (img_size // patch_size) ** 2
        self.cls_token = nn.Parameter(torch.zeros(1, 1, d_model))
        self.pos_embed = nn.Parameter(torch.zeros(1, num_patches + 1, d_model))

    def forward(self, x):  # x: (B, 3, 224, 224)
        x = self.proj(x)  # (B, d_model, 14, 14)
        x = x.flatten(2).transpose(1, 2)  # (B, 196, d_model)
        cls = self.cls_token.expand(x.size(0), -1, -1)
        x = torch.cat([cls, x], dim=1)  # (B, 197, d_model)
        x = x + self.pos_embed
        return x

Real products, models, and research that use this idea.

OpenAI CLIP uses a ViT image encoder paired with a text encoder for joint image-text contrastive learning.
Meta DINOv2 (released 2024) trains ViT self-supervised on internet-scale image data, producing the standard pretrained image backbone for downstream tasks.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat happens when you run a ViT trained at 224 x 224 on a 384 x 384 image?

The patch count grows (576 instead of 196), so the learned position embeddings (which have shape (1, 197, d_model)) don't fit. The standard fix is to interpolate the position embeddings to the new grid, treating them as a 14 x 14 grid of vectors and resizing to a 24 x 24 grid via bilinear or bicubic interpolation. Most production ViT codebases include this resize logic for variable-resolution inference.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Using a causal mask for ViT. Images have no temporal direction, all patches exist simultaneously, so attention is fully bidirectional. Causal masking would arbitrarily block patches from seeing later patches in the (artificial) raster order.

60 second bullets to scan on the way to the call.

Name each step of the ViT pipeline that turns a raw image into a token sequence
Typical patch size and resulting sequence length for 224 x 224 input

Primary sources. Browse if you want the original framing.