What's the practical tradeoff between learned and sinusoidal positional embeddings?

Attempt it

Quick reference

TL;DR

Learned PE: trainable per position embeddings; flexible and often slightly better in distribution; HARD CAP at max training position.

Memory aid

Easy to grasp

Imagine a theater that prints custom seat covers for every seat, row 1 seat A gets one design, row 1 seat B gets another, all the way through row 100. The theater learned which covers belong on which seats by trial and error. Now compare that to a different theater that uses a mathematical formula to color every seat, given a row and seat number, the formula spits out a color pattern, even for seats that haven't been built yet. The first theater (learned positional encoding) makes seats feel exactly right when you're inside the building it was trained on, but if you walk into row 101, there's no cover for that seat. The second theater (sinusoidal) has a cover for any seat number you can name, though the patterns aren't quite as snug for the seats it's actually seen.

Key concepts

Learned positional embedding Sinusoidal PE Positional encoding

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

Walk the parametric vs nonparametric framing, give the hard cap on learned PE consequence (BERT's 512-token limit), note sinusoidal's weak in practice generalization, and explain why RoPE supersedes both.

Side-by-side

Property	Learned PE	Sinusoidal PE	RoPE
Trainable parameters	max_position × d_model	None	None
Defined at any position?	No (hard cap)	Yes (formula)	Yes (but angles cycle)
Bounded values?	Usually (no guarantee)	Yes (-1, 1)	Q, K norms unchanged
Structural smoothness	None (each vector independent)	Strong (sin/cos)	Strong (rotation)
In distribution fit	Often slightly best	Slightly weaker	Matches or beats both
Length extension	Requires retraining/interp	Weak generalization	PI / YaRN

Real-world examples

Real products, models, and research that use this idea.

Original BERT used learned PE with max_position=512, and this is exactly why long document NLP became its own subfield in 2019-2020.
GPT-1 and GPT-2 used learned PE; GPT-2 had max_position=1024.
The original Transformer and T5 use sinusoidal-style PE.
Llama, Mistral, Qwen, DeepSeek, GPT-J, modern decoder LLMs all use RoPE, not learned or sinusoidal PE.
BLOOM and MPT use ALiBi instead of any input-side PE.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat goes wrong concretely when you feed a learned-PE model a position beyond its training max?

QWhy is sinusoidal PE's in-principle generalization to longer sequences weak in practice?

QWhy does RoPE replace both learned and sinusoidal PE rather than just augmenting them?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Confusing 'fewer FLOPs' or 'numerical stability' with the actual tradeoff, the real distinction is parametric vs nonparametric, with the consequent hard cap on learned PE.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.