Walk through the sinusoidal positional encoding from Vaswani 2017. What's the formula? What mathematical property does the construction try to give the model, and how (sketch the argument)?
Sinusoidal PE gives each position a fixed sin/cos fingerprint with geometrically spaced frequencies, added to the input embedding.
Imagine giving every word a fingerprint built from many tiny clocks. Some clocks tick once per word. Others tick once per hundred words. The slowest ones tick about once per ten thousand words. Two words next to each other have almost the same set of clock readings. Two words far apart have very different ones. The model can compare fingerprints and figure out how far apart any two words are, without anyone ever writing down a word number. Sine on even slots and cosine on odd slots make a position shift correspond to a simple, repeatable transformation of the fingerprint.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Sinusoidal PE is the historical baseline for transformer positional encoding and the easiest scheme to derive from first principles. Vaswani et al. 2017 introduced it in the original Transformer paper:
The geometric spacing of frequencies, the sin/cos pairing on adjacent dimensions, and the base 10000 are all design choices with specific motivations.
The formula and what each piece does
Even dimensions get sin(pos / 10000^(2i/d)). Odd dimensions get the cosine of the same argument. The base 10000 is a convention from the paper. Bigger base spreads frequencies wider.
The angular frequency at dimension i is ω_i = 10000^(-2i/d). With d = 512:
- At i = 0:
ω = 1, period ≈ 6.28 positions. The sinusoid oscillates rapidly and resolves single position differences. - At i = 128:
ω ≈ 0.01, period ≈ 628 positions. The sinusoid carries medium range positional structure. - At i = 255 (the last pair):
ω ≈ 0.0001, period ≈ 61000 positions. The sinusoid barely changes across a 10k token sequence and carries only coarse positional class.
Different dimensions resolve position at different scales. High frequency dims tell apart adjacent positions. Low frequency dims hold the coarse 'beginning vs middle vs end' position class. The geometric spacing across orders of magnitude is what lets d_model handle both short and long range position structure with the same fixed size vector.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Aspect | Sinusoidal PE | Learned PE | RoPE |
|---|---|---|---|
| Where applied | Added to input embedding | Added to input embedding | Rotation on Q, K |
| Trainable parameters | None | One vector per position | None |
| Relative position property | Indirect (must be learned) | Indirect (must be learned) | Direct (built into QKᵀ) |
| Defined at any position? | Yes (math formula) | No (hard max cap) | Yes (but angles cycle) |
| Practical length generalization | Weak | None | Weak; PI/YaRN fixes |
Real products, models, and research that use this idea.
- The original Transformer used sinusoidal PE in the encoder and decoder.
- T5 drops absolute PE entirely and uses only a learned, bucketed relative attention bias, a common alternative to sinusoidal PE.
What an interviewer would ask next. Try answering before peeking at the approach.
QShow why PE(pos+k) is a fixed linear transformation of PE(pos).
Each (sin(ω_i·pos), cos(ω_i·pos)) pair transforms under a 2x2 rotation R(ω_i·k) when you shift position by k (angle addition identities). Stacking pairs gives a block diagonal rotation M_k that depends only on k, not pos.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Confusing the linear combination property (a fact about sin/cos) with a CLAIM that the model automatically uses relative positions. The property only enables relative position behavior. The model still has to learn to extract it from the additive sum.
60 second bullets to scan on the way to the call.
The sin and cos formula with even and odd dim pairing
Where PE enters the model relative to attention
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.