Trained heads visibly specialize (syntactic, positional, induction, rare token), but many are post hoc prunable, specialization and redundancy both hold.
Imagine handing one murder case to twelve detectives. They don't all duplicate each other's notes. One focuses on fingerprints, another on alibis, another on motive. Once the case is solved, you could have done it with fewer detectives, but during the investigation, you needed the whole team so the right specialization could emerge naturally. Attention heads work the same way: during training, they each gravitate toward a different relationship type in the data. After training, many of them are doing similar enough work that you can drop a chunk without losing much quality.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Interpretability research since 2019 has built a steady picture of what attention heads actually do in trained transformers. The frame that holds up across architectures and scales has three layers: specialization, mechanistic circuits, and redundancy.
Each layer answers a different question. Specialization tells you what heads do. Mechanistic circuits tell you how heads compose into algorithms. Redundancy tells you which heads are dispensable at inference. Conflating the three is where most beginner confusion comes from, and where most online hot-takes about 'multi-head doesn't matter' originate.
The stakes are real beyond academic interpretability. Modern KV-cache compression (GQA, MQA, MLA), pruning, distillation, and architecture-design decisions all rest on a correct read of what heads contribute. We'll walk all three layers, then close on how the picture shapes 2026 architecture choices in Llama 4, Mistral Large 3, DeepSeek V4, and Gemma 4.
Specialization is real and emergent
Voita 2019 (Analyzing Multi-Head Self-Attention, 1905.09418) looked at translation encoders and found heads cleanly fall into a small taxonomy. Positional heads attend to the previous or next token regardless of content. Syntactic heads attend along dependency relations like subject to verb or noun to adjective. Rare-token heads focus on uncommon words.
The methodology was direct: visualize attention matrices, then check whether each head's pattern matches a known linguistic relation better than chance. Across 48 heads in a 6-layer encoder, roughly 30-40% had clear functional labels; the rest were either redundant copies or had patterns the team could not classify with high confidence.
Clark 2019 (What Does BERT Look At?, 1906.04341) ran the same kind of analysis on BERT and found heads tracking dependency relations the model never saw labeled, direct object, prepositional object, coreference. The matches were better than random parser baselines on real corpora, which means the structure was discovered, not memorized.
The key word is emergent. The architecture enabled specialization, parallel softmaxes can carry independent patterns without trading off softmax mass, but gradient descent found the actual roles. Nothing in the training loss said 'attend to syntactic dependencies.' That structure fell out of next token prediction on natural text. The architecture provides the substrate; the data provides the inductive pressure.
Specialization is not a design feature. It is a learned consequence of running gradient descent on natural language with parallel attention heads.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Voita et al. 2019 identified 'positional', 'syntactic', and 'rare token' head types in Transformer NMT encoders.
- Clark et al. 2019 found BERT heads tracking dependency relations like direct-object and prepositional-object without explicit parse supervision.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy do induction heads emerge sharply during training rather than gradually?
Olsson et al. argue induction heads require two heads working in series (one for matching the current token to past occurrences, one for copying the following position). The two head dependency creates a threshold effect: until both pieces are in place, neither is useful and gradient pressure is weak. Once both emerge, the circuit becomes useful and gets reinforced rapidly, hence the sharp phase transition rather than a gradual ramp.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Asserting either that all heads are identical (wrong) or that no heads are redundant (also wrong). The interesting truth is both specialization and redundancy coexist.
60 second bullets to scan on the way to the call.
Specialization vs redundancy and how both can be real
Two named specialization types (positional, syntactic, induction, coreference, rare token)
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.