Walk through the sinusoidal positional encoding from Vaswani 2017. What's the formula? What mathematical property does the construction try to give the model, and how (sketch the argument)?
Sinusoidal PE gives each position a fixed sin/cos fingerprint with geometrically spaced frequencies, added to the input embedding.
Imagine giving every word a fingerprint built from many tiny clocks. Some clocks tick once per word. Others tick once per hundred words. The slowest ones tick about once per ten thousand words. Two words next to each other have almost the same set of clock readings. Two words far apart have very different ones. The model can compare fingerprints and figure out how far apart any two words are, without anyone ever writing down a word number. The choice of having sine on even slots and cosine on odd slots is what makes shifting positions correspond to a simple, repeatable transformation of the fingerprint.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Sinusoidal PE is the historical baseline for transformer positional encoding and the easiest scheme to derive from first principles. Vaswani et al. 2017 introduced it in the original Transformer paper:
The construction is more thoughtful than it looks. The geometric spacing of frequencies, the sin/cos pairing on adjacent dimensions, the base 10000, all of these are design choices with specific motivations. Understanding why is what separates a textbook recap from a senior level grasp of how positional encoding actually works.
This deep dive walks the formula piece by piece, derives the angle addition property that makes the construction relative position friendly, explains why that property is indirect (and what RoPE changed to make it structural), and covers the practical properties that kept sinusoidal PE relevant in some model families even after RoPE took over the decoder frontier.
The formula and what each piece does
Even dimensions get sin(pos / 10000^(2i/d)). Odd dimensions get the cosine of the same argument. The base 10000 is a convention from the paper, chosen so the lowest frequency completes about one cycle across a 10k position sequence. Bigger base spreads frequencies wider.
The angular frequency at dimension i is ω_i = 10000^(-2i/d). With d = 512:
- At i = 0:
ω = 1, period ≈ 6.28 positions. The sinusoid oscillates rapidly and resolves single position differences. - At i = 128:
ω ≈ 0.1, period ≈ 63 positions. Medium range positional structure. - At i = 256:
ω ≈ 0.0001, period ≈ 62000 positions. The sinusoid barely changes across a 10k token sequence and carries only coarse positional class.
Different dimensions resolve position at different scales. High frequency dims tell apart adjacent positions. Low frequency dims hold the coarse 'beginning vs middle vs end' position class. The geometric spacing across orders of magnitude is what lets d_model simultaneously handle both short and long range position structure with the same fixed size vector.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Aspect | Sinusoidal PE | Learned PE | RoPE |
|---|---|---|---|
| Where applied | Added to input embedding | Added to input embedding | Rotation on Q, K |
| Trainable parameters | None | One vector per position | None |
| Relative position property | Indirect (must be learned) | Indirect (must be learned) | Direct (built into QKᵀ) |
| Defined at any position? | Yes (math formula) | No (hard max cap) | Yes (but angles cycle) |
| Practical length generalization | Weak | None | Weak; PI/YaRN fixes |
Real products, models, and research that use this idea.
- The original Transformer used sinusoidal PE in the encoder and decoder.
- T5 uses sinusoidal style position info combined with a learned relative attention bias.
What an interviewer would ask next. Try answering before peeking at the approach.
QShow why PE(pos+k) is a fixed linear transformation of PE(pos).
Each (sin(ω_i·pos), cos(ω_i·pos)) pair transforms under a 2x2 rotation R(ω_i·k) when you shift position by k (angle addition identities). Stacking pairs gives a block diagonal rotation M_k that depends only on k, not pos.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Confusing the linear combination property (a fact about sin/cos) with a CLAIM that the model automatically uses relative positions, the property only enables relative position behavior; the model still has to learn to extract it from the additive sum.
60 second bullets to scan on the way to the call.
The sin and cos formula with even and odd dim pairing
Where PE enters the model relative to attention
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.