Walk through Vaswani 2017's sinusoidal positional encoding, what's the formula and what property does it give?

**Sinusoidal PE** is the original positional encoding from Vaswani 2017. For a token at position pos and dimension i: $$\text{PE}_{(pos,2i)} = \sin\!\left(\frac{pos}{10000^{2i/d}}\right)$$ Odd dimensions use cosine of the same argument. Together they form a `d_model`-dim fingerprint for the position, added to the input embedding before the first transformer block. The frequency depends on the dimension. At i=0 the sinusoid oscillates rapidly and tells apart adjacent positions. At i = d/2 the argument is pos/10000, so the sinusoid barely changes across realistic sequences and encodes coarse position class. Frequencies span many orders of magnitude geometrically. The property the construction targets is **relative position friendliness**. By angle addition identities, `PE(pos + k) = M_k · PE(pos)` where M_k depends only on k. W_q and W_k are linear and can in principle compose with M_k to detect relative offsets, but they have to learn this. Wins: bounded values, zero parameters, defined at any position. Catch: extrapolation is weak because the model only learns the property over the training range. Modern LLMs use RoPE instead.

Formula first: $$\text{PE}_{(pos,2i)} = \sin\!\left(\frac{pos}{10000^{2i/d}}\right)$$ and `PE(pos, 2i+1) = cos(...)`. Even dims sine, odd dims cosine. Frequencies geometric from 1 down to about `1/10000`. The full d-dim PE vector is added to the input embedding before the first transformer block. **The property the construction targets** is relative position friendliness. By angle addition identities, `PE(pos + k)` is a fixed linear transform of `PE(pos)` with coefficients depending only on k, not on pos. Stack the 2D blocks and `PE(pos + k) = M_k · PE(pos)`. Since W_q and W_k are linear, they can in principle compose with M_k to detect relative offsets. But the property is **indirect**. The model still has to learn the projection that extracts relative position from the additive `(content + PE)` mix. It learns this over the training position range and only over that range, which is why empirical length generalization is weak. The wins are bounded values in `[-1, 1]`, zero trainable parameters, and definition at any position. Mostly replaced by RoPE in modern decoder LLMs because RoPE makes the same property structural rather than learnable.