What has interpretability research shown about individual attention heads in trained transformers?

Interpretability research since 2019 gives a clear picture: heads specialize visibly and meaningfully, even though there is also real redundancy. Both findings are true at the same time. **Voita 2019** looked at translation encoders and found heads consistently fall into a small taxonomy: positional heads (attend to previous or next token), syntactic heads (attend along grammar relations), rare token heads. None of these were supervised; they emerged from standard training. **Clark 2019** ran the same analysis on BERT and found heads tracking dependency relations like direct object and coreference. BERT never saw parse trees, yet specific heads matched parser output better than random baselines. **Olsson 2022** identified **induction heads**, pairs of heads in successive layers implementing a copying algorithm: find a previous occurrence of the current token, attend to what came after. They are the substrate for in context learning and emerge sharply during training alongside ICL capability. **Michel 2019** showed you can prune many heads post-training with minimal quality loss. This sounds contradictory but is not. Training needs head diversity to discover the specializations; once found, multiple heads end up covering similar work and a chunk is dispensable at inference.

Trained heads visibly specialize, and that's one of the most reproducible findings in mechanistic interpretability. Voita 2019 and Clark 2019 cataloged positional, syntactic, and rare token heads in encoder transformers and BERT. Olsson 2022 went further and identified **induction heads**, pairs of heads in successive layers that implement in context copying and emerge sharply alongside in context learning capability. None of this was designed in. It emerged from gradient descent on the standard objective, which is why people care about it. The complication is Michel 2019, who showed many heads can be pruned post-training with little quality loss. That's not a contradiction, it's a separate observation. Training needs head diversity to find the specializations; inference can often get by with fewer. Both findings are real, and they sit at different points in the model's lifecycle.