Click any words you think contain an error. Click again to unmark.
Mark at least one word to submit.
The flaw is the central claim itself. Attention does NOT automatically capture positional information. It is permutation-invariant: if you shuffle the input tokens (without changing the embeddings), the attention outputs are the same vectors in the same shuffled order.
Here is why. Self-attention computes Q, K, V as linear projections of token embeddings: `Q = X W_Q`, `K = X W_K`, `V = X W_V`. The attention scores are `softmax(Q K^T / sqrt(d_k))`. Nothing in this computation references the position index i or j. The dot product `q_i . k_j` depends only on the content of the two embeddings, not on where they sit in the sequence.
The 'attend more to nearby keys' behavior the speaker describes is real, but it appears only AFTER positional encoding is added. Sinusoidal encoding (Vaswani 2017), learned encoding, RoPE (rotary), and ALiBi (linear bias) all inject position information into the model, either by adding it to embeddings or by modifying the attention scores directly. Training then teaches the model to use that positional signal, and the locality patterns emerge.
Remove the positional encoding entirely and attention becomes a set operation: it does not know that position 5 comes after position 4 or that nearby tokens are usually more relevant. The 'small refinement' framing in the claim has the priority backwards: positional encoding is the source of all positional awareness, not a refinement on top of it.
The claim is wrong at its foundation. Attention is permutation-invariant; the only positional awareness in a transformer comes from the positional encoding.
The permutation argument is direct. Take input tokens x_1, ..., x_n with embeddings e_1, ..., e_n. Self-attention computes scores `s_ij = q_i . k_j / sqrt(d_k)` where q_i and k_j are linear functions of e_i and e_j alone. The indices i and j do not appear in the formula except as labels. If you permute the input tokens, you get the same outputs in the same permuted order:
$$f(\pi(x)) = \pi(f(x))$$
No positional bias is created by attention itself. The 'attend more to nearby keys' behavior the speaker mentions is real in trained models, but it is a learned consequence of positional encoding plus training data, not a structural property of attention.
The four positional encoding schemes you see in 2026 production all inject position somewhere:
- Sinusoidal: added to embeddings before the first layer.
- Learned: added to embeddings (learned table per position).
- RoPE: rotates Q and K vectors by position-dependent angles before the dot product. Used by Llama, Mistral, Qwen, DeepSeek.
- ALiBi: adds a linear distance penalty to attention scores directly.
Without any of these, attention is a set operation. Position is an architectural injection, not a free property.
The 'small refinement' framing in the claim is the most important error. Position is not refinement, it is the entire source of order awareness in the transformer.
Quick reference
TL;DR
The flaw is the premise. Attention has zero positional awareness on its own; it is permutation-invariant. Locality comes from positional encoding plus training, not from attention itself.
Imagine a librarian who can fetch any book from a shelf in equal time no matter where it sits. Now blindfold the librarian. Without any markers on the shelves, the librarian has no idea which book is at position 1 versus position 100, all the shelf does is hold books and offer them up on request. Attention is that blindfolded librarian. The 'shelf positions' (positional encoding) are the markers we add so the librarian can reason about order at all.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Strictly bag of words tasks where token order is irrelevant If the task does not depend on order, positional encoding is not strictly necessary and the model can learn from content alone.
When debugging a position-dependent failure If a model fails on long-context tasks, the positional encoding scheme (sinusoidal vs RoPE vs ALiBi) is often the right place to start rather than the attention mechanism itself.
Cross-attention with structured inputs that carry their own position Some cross-attention setups inject position through the key side via a separate mechanism; the 'attention captures position' claim still does not hold, but the injection point differs.
When comparing attention variants that differ in positional handling Conflating attention pattern (full vs sliding) with positional encoding (RoPE vs ALiBi) is a common confusion; they are independent axes.
Quick scan · 2–4 min
Everything important, quickly.
4m: why attention is permutation-invariant by construction, the four positional encoding schemes and where each injects, why trained locality patterns come from encoding plus training, and the model failure mode if positional encoding is absent.
Real-world examples
Real products, models, and research that use this idea.
1Vaswani et al. 2017 introduced sinusoidal positional encoding precisely because attention has no positional awareness.
2Llama 4 Maverick, Mistral Large 3, Qwen 3.5, and DeepSeek V4 all use RoPE for positional encoding.
3BLOOM and MPT used ALiBi for length extrapolation, demonstrating that the positional choice is what enables extrapolation, not attention.
4Vision Transformers (ViT) require positional encoding for image patches because attention treats patches as a set.
5Ablation studies in the original Transformer paper showed that removing positional encoding causes catastrophic degradation on sequence tasks.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does RoPE give the attention dot product a relative-position property?
Sign in or go Pro to see the approach for this follow-up.
QHow does ALiBi's per-head slope schedule produce its length-extrapolation property?
Sign in or go Pro to see the approach for this follow-up.
QWhat happens if you train a transformer with no positional encoding on a sequence task?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Reading attention's locality-friendly behavior in trained models as evidence that attention itself encodes position. The locality comes from the positional encoding plus training; remove the encoding and the behavior vanishes.