Same topic, related formats. Practice these next.
Walk through the sinusoidal positional encoding from Vaswani 2017. What's the formula? What mathematical property does the construction try to give the model, and how (sketch the argument)?
Sinusoidal PE gives each position a fixed sin/cos fingerprint with geometrically spaced frequencies, added to the input embedding.
Imagine giving every word a fingerprint built from many tiny clocks. Some clocks tick once per word. Others tick once per hundred words. The slowest ones tick about once per ten thousand words. Two words next to each other have almost the same set of clock readings. Two words far apart have very different ones. The model can compare fingerprints and figure out how far apart any two words are, without anyone ever writing down a word number. The choice of having sine on even slots and cosine on odd slots is what makes shifting positions correspond to a simple, repeatable transformation of the fingerprint.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
Walk the formula with geometric frequencies, explain the angle addition / linear transformation property, note it's INDIRECT (model must learn), state bounded + zero param + defined at any pos benefits, and contrast with RoPE's direct relative position property.
| Aspect | Sinusoidal PE | Learned PE | RoPE |
|---|---|---|---|
| Where applied | Added to input embedding | Added to input embedding | Rotation on Q, K |
| Trainable parameters | None | One vector per position | None |
| Relative position property | Indirect (must be learned) | Indirect (must be learned) | Direct (built into QKᵀ) |
| Defined at any position? | Yes (math formula) | No (hard max cap) | Yes (but angles cycle) |
| Practical length generalization | Weak | None | Weak; PI/YaRN fixes |
Real products, models, and research that use this idea.
What an interviewer would ask next. Try answering before peeking at the approach.
Red flags and common mistakes that signal junior thinking. Click to expand.
Confusing the linear combination property (a fact about sin/cos) with a CLAIM that the model automatically uses relative positions, the property only enables relative position behavior; the model still has to learn to extract it from the additive sum.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.