Spot the error in this claim about attention and position.

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

Quick reference

TL;DR

Both sentences are wrong. Attention is permutation-equivariant, it has no inherent notion of 'near' or 'far' positions. Positional encoding is mandatory.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine someone hands you a stack of index cards in random order and asks 'which cards were neighbors in the original stack?' If the cards don't have page numbers written on them, the question is unanswerable: the stack lost its order the moment you grabbed it. Attention is exactly that grabber: it sees the cards as a pile, never as a stack. The claim that 'nearby cards score higher together' is just false until someone writes a position number on each one.

Key concepts

Permutation invariance / equivariance Positional encoding Common transformer misconception

Quick scan · 2–4 min

Everything important, quickly.

5 min: diagnose both errors via the equivariance property, separate content-based scoring from position-aware behavior, walk through Deep Sets framing as the formal reason PE is non-optional, and address the causal-mask counter-argument from Haviv 2022.

Side-by-side

Claim	Truth value	Why
Attention captures position implicitly	False	Permutation-equivariance; scores are content-based
Adjacent tokens get high attention weights structurally	False	Weights are content match, not distance
Transformers need positional encoding	True	Otherwise the model is a set function
Causal masking provides some position info	Partially true	Only in causal decoders; doesn't help encoders

Real-world examples

Real products, models, and research that use this idea.

Haviv et al. 2022 'Transformer Language Models without Positional Encodings Still Learn Positional Information': the canonical NoPE study, which nonetheless found NoPE underperforms RoPE at scale.
Kazemnejad et al. 2023 'The Impact of Positional Encoding on Length Generalization': empirical comparison of PE schemes.
The original Transformer paper, section 3.5, explicitly motivates the need for positional encoding.
Every modern frontier LLM (Llama 4 Maverick, DeepSeek V4, Gemini 3.1 Pro, GPT-5.5, Claude Opus 4.7) uses RoPE or a variant: none rely on implicit position handling.
Set Transformer deliberately keeps the equivariance and adds a pooling step instead, confirming that PE-free attention really is a set operation.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QIf you train a transformer without PE on a synthetic task where the answer only depends on the multiset of input tokens (no order), does it work?

Sign in or go Pro to see the approach for this follow-up.

QWhat's the strongest argument FOR the 'attention captures position implicitly' view, and how do you rebut it?

Sign in or go Pro to see the approach for this follow-up.

QHow does the Deep Sets framework help you predict what tasks a NoPE transformer will fail on?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.