Spot the flaw: a claim that attention by itself notices position.

The flaw is the central claim itself. Attention does NOT automatically capture positional information. It is permutation-invariant: if you shuffle the input tokens (without changing the embeddings), the attention outputs are the same vectors in the same shuffled order. Here is why. Self-attention computes Q, K, V as linear projections of token embeddings: `Q = X W_Q`, `K = X W_K`, `V = X W_V`. The attention scores are `softmax(Q K^T / sqrt(d_k))`. Nothing in this computation references the position index i or j. The dot product `q_i . k_j` depends only on the content of the two embeddings, not on where they sit in the sequence. The 'attend more to nearby keys' behavior the speaker describes is real, but it appears only AFTER positional encoding is added. Sinusoidal encoding (Vaswani 2017), learned encoding, RoPE (rotary), and ALiBi (linear bias) all inject position information into the model, either by adding it to embeddings or by modifying the attention scores directly. Training then teaches the model to use that positional signal, and the locality patterns emerge. Remove the positional encoding entirely and attention becomes a set operation: it does not know that position 5 comes after position 4 or that nearby tokens are usually more relevant. The 'small refinement' framing in the claim has the priority backwards: positional encoding is the source of all positional awareness, not a refinement on top of it.

The claim is wrong at its foundation. Attention is permutation-invariant; the only positional awareness in a transformer comes from the positional encoding. The permutation argument is direct. Take input tokens x_1, ..., x_n with embeddings e_1, ..., e_n. Self-attention computes scores `s_ij = q_i . k_j / sqrt(d_k)` where q_i and k_j are linear functions of e_i and e_j alone. The indices i and j do not appear in the formula except as labels. If you permute the input tokens, you get the same outputs in the same permuted order: $$f(\pi(x)) = \pi(f(x))$$ No positional bias is created by attention itself. The 'attend more to nearby keys' behavior the speaker mentions is real in trained models, but it is a learned consequence of positional encoding plus training data, not a structural property of attention. The four positional encoding schemes you see in 2026 production all inject position somewhere: - Sinusoidal: added to embeddings before the first layer. - Learned: added to embeddings (learned table per position). - RoPE: rotates Q and K vectors by position-dependent angles before the dot product. Used by Llama, Mistral, Qwen, DeepSeek. - ALiBi: adds a linear distance penalty to attention scores directly. Without any of these, attention is a set operation. Position is an architectural injection, not a free property. The 'small refinement' framing in the claim is the most important error. Position is not refinement, it is the entire source of order awareness in the transformer.