Each token id indexes one row of the [vocab, d_model] embedding table; the positional vector is added (not concatenated) before the first block sees it.
Imagine the embedding table as a giant phone book with one row per word in your dictionary. The tokenizer turns the sentence into a list of phone numbers (integer ids), and each id tells the model which row to copy out. That row is a fixed-length list of numbers that represents the meaning of the word. But the model also needs to know where each word sits in the sentence, so it grabs a second little vector that encodes position. The two vectors have the same length and the model simply adds them together, number by number, like overlaying two transparencies. The resulting vector enters block 1 carrying both what the token is and where it lives in the line.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The first thing a transformer does to your text is the most boring step in the whole model, and it is also the only step where a discrete object (an integer token id) becomes a continuous vector that the rest of the network can do calculus on. Understanding this step is what stops people from believing in transformer magic. There is no magic. There is a giant lookup table, a small piece of addition, and then everything afterwards is linear algebra plus a handful of nonlinearities.
This walkthrough traces a single token from its integer id to the vector that enters block 1, for the classic RoPE-free design used by GPT-2, BERT, and many small or mid-2020s models. We will also note where RoPE-era models diverge, because that is the modern default and you should know the difference.
Mental model: the embedding lookup is a phone book with
vocab_sizerows. The position table is a parallel phone book withmax_seq_lenrows. Both phone books have the same number of columns (d_model). The model copies one row from each and adds them.
The embedding matrix and the row lookup
The model holds a single learned matrix E of shape [vocab_size, d_model]. For a Llama-2 7B style model that might be [32000, 4096], totalling about 131M parameters. Every row of this matrix is the model's representation of one vocabulary entry: the row for the lives near the row for a because they play similar grammatical roles, the row for Paris lives near the row for London, and so on. These geometries are not designed; they emerge from training.
When the tokenizer returns an id i, the model produces E[i], a vector of length d_model. Frameworks implement this as a gather operation on the GPU: it reads one row of memory, no multiplication, no nonlinearity. The math is equivalent to multiplying a one-hot row vector by E, but you would never compute it that way for a real vocabulary, because a [B, T, 32000] one-hot tensor is huge and wasteful.
The gradient flow matters here too. When the loss propagates back to the embedding lookup, only the rows whose ids appeared in the batch receive a gradient. The other roughly 99.9% of rows sit idle that step. That sparsity is why embedding parameters are sometimes treated specially by optimisers (Adam with sparse updates, separate learning rates, etc.).
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Llama 3 and Llama 4 use RoPE inside attention and have no separate learned positional embedding table; the input to block 1 is just `E[tok]`.
- BERT and GPT-2 used learned positional embeddings added to token embeddings, the textbook recipe this question describes.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is the embedding lookup typically implemented as a gather rather than a dense matmul?
Sparsity of the gradient (only touched rows get a gradient), memory bandwidth for moving a [B, T, V] one-hot tensor, and integer indexing being a native GPU op.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Thinking the positional vector is concatenated, or that the lookup returns a column. The table is row-major: ids index rows, and positions are summed elementwise.
60 second bullets to scan on the way to the call.
Shape of the embedding matrix and what indexing it returns
Shape of a learned positional table in a RoPE-free model
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.