Spot the error in this claim about attention and position.
Click any words you think contain an error. Click again to unmark.
Both sentences are wrong. Attention is permutation-equivariant, it has no inherent notion of 'near' or 'far' positions. Positional encoding is mandatory.
Imagine someone hands you a stack of index cards in random order and asks 'which cards were neighbors in the original stack?' If the cards don't have page numbers written on them, the question is unanswerable: the stack lost its order the moment you grabbed it. Attention is exactly that grabber: it sees the cards as a pile, never as a stack. The claim that 'nearby cards score higher together' is just false until someone writes a position number on each one.
Detailed answer & concept explanation~4 min readEverything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Everything important, quickly.
5 min: diagnose both errors via the equivariance property, separate content-based scoring from position-aware behavior, walk through Deep Sets framing as the formal reason PE is non-optional, and address the causal-mask counter-argument from Haviv 2022.
| Claim | Truth value | Why |
|---|---|---|
| Attention captures position implicitly | False | Permutation-equivariance; scores are content-based |
| Adjacent tokens get high attention weights structurally | False | Weights are content match, not distance |
| Transformers need positional encoding | True | Otherwise the model is a set function |
| Causal masking provides some position info | Partially true | Only in causal decoders; doesn't help encoders |
Real products, models, and research that use this idea.
- Haviv et al. 2022 'Transformer Language Models without Positional Encodings Still Learn Positional Information': the canonical NoPE study, which nonetheless found NoPE underperforms RoPE at scale.
- Kazemnejad et al. 2023 'The Impact of Positional Encoding on Length Generalization': empirical comparison of PE schemes.
- The original Transformer paper, section 3.5, explicitly motivates the need for positional encoding.
- Every modern frontier LLM (Llama 4 Maverick, DeepSeek V4, Gemini 3.1 Pro, GPT-5.5, Claude Opus 4.7) uses RoPE or a variant: none rely on implicit position handling.
- Set Transformer deliberately keeps the equivariance and adds a pooling step instead, confirming that PE-free attention really is a set operation.
What an interviewer would ask next. Try answering before peeking at the approach.
QIf you train a transformer without PE on a synthetic task where the answer only depends on the multiset of input tokens (no order), does it work?
QWhat's the strongest argument FOR the 'attention captures position implicitly' view, and how do you rebut it?
Don't say thisRed flags and common mistakes that signal junior thinking. Click to expand.
Red flags and common mistakes that signal junior thinking. Click to expand.
Believing attention has any built-in notion of distance between tokens. It doesn't. Distance enters only via positional encoding.
The night-before-the-interview bullets. Scan these on the way to the call.
Primary sources. Skim if you want the original framing.
Same topic, related formats. Practice these next.