What's the practical tradeoff between learned and sinusoidal positional embeddings?
Learned PE: trainable per position embeddings; flexible and often slightly better in distribution; HARD CAP at max training position.
Imagine a theater that prints custom seat covers for every seat, row 1 seat A gets one design, row 1 seat B gets another, all the way through row 100. The theater learned which covers belong on which seats by trial and error. Now compare that to a different theater that uses a mathematical formula to color every seat, given a row and seat number, the formula spits out a color pattern, even for seats that haven't been built yet. The first theater (learned positional encoding) makes seats feel exactly right when you're inside the building it was trained on, but if you walk into row 101, there's no cover for that seat. The second theater (sinusoidal) has a cover for any seat number you can name, though the patterns aren't quite as snug for the seats it's actually seen.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Learned vs sinusoidal positional embeddings is the classic parametric vs nonparametric tradeoff applied to a concrete part of the transformer architecture: how a token at position t gets a vector that says 'I am at position t.'
Both schemes add their per position vector to the input embedding before the first attention layer. The mechanism is identical; the source of the vector differs. Learned PE pulls from a trainable table; sinusoidal PE evaluates a closed form formula.
This deep dive walks both schemes, the consequences of their structural differences, and why the field largely moved past both to RoPE and ALiBi.
Mental model: the differences only matter when you push past
max_position. In distribution they're nearly interchangeable; out of distribution they fail differently.
Learned PE: parametric flexibility with a hard cap
How it works
Maintain a trainable embedding table PE: (max_position, d_model). For each input token at position t, look up PE[t] and add to the token embedding.
self.pe = nn.Embedding(max_position, d_model)
x = token_embed(tokens) + self.pe(positions)
What you gain
- Parametric flexibility. Each position vector is shaped independently by gradient descent. In distribution fit is often slightly better than sinusoidal because the parameterization is strictly more flexible.
- No formula to hand-tune. The model learns whatever positional features it needs.
What you lose
- Hard cap at
max_position. Positionmax_position+1literally has no embedding row. - No smoothness between rows. Each position vector is independent; there's no inductive bias that positions 511 and 512 should be 'close.'
- Memory scales with context. A 32k context with
d_model=4096needs a(32k × 4k)PE table, 128M parameters just for positional info.
Canonical example
BERT's 512-token limit. The MLM training was done at max_position=512, and feeding longer documents requires resizing the PE table and fine-tuning, which is why long document NLP became its own subfield in 2019-2020 (Longformer, BigBird, etc.).
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Property | Learned PE | Sinusoidal PE | RoPE |
|---|---|---|---|
| Trainable parameters | max_position × d_model | None | None |
| Defined at any position? | No (hard cap) | Yes (formula) | Yes (but angles cycle) |
| Bounded values? | Usually (no guarantee) | Yes (-1, 1) | Q, K norms unchanged |
| Structural smoothness | None (each vector independent) | Strong (sin/cos) | Strong (rotation) |
| In distribution fit | Often slightly best | Slightly weaker | Matches or beats both |
| Length extension | Requires retraining/interp | Weak generalization | PI / YaRN |
Real products, models, and research that use this idea.
- Original BERT used learned PE with max_position=512, and this is exactly why long document NLP became its own subfield in 2019-2020.
- GPT-1 and GPT-2 used learned PE; GPT-2 had max_position=1024.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat goes wrong concretely when you feed a learned-PE model a position beyond its training max?
Two cases. (1) If the code clips to max_position, every position past the cap collapses to the same embedding, and the model sees a sequence with many tokens at 'position 512': meaningless. (2) If the code doesn't clip, it's a literal index out of bounds and crashes. Either way, extension requires resizing the PE table and fine-tuning.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Confusing 'fewer FLOPs' or 'numerical stability' with the actual tradeoff, the real distinction is parametric vs nonparametric, with the consequent hard cap on learned PE.
60 second bullets to scan on the way to the call.
Learned PE: trainable per position embedding table
Sinusoidal PE: closed form sin/cos with geometric frequencies
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.