What has interpretability research shown about individual attention heads in trained transformers?

Attempt it

Quick reference

TL;DR

Trained heads visibly specialize (syntactic, positional, induction, rare token), but many are post hoc prunable, specialization and redundancy both hold.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine handing one murder case to twelve detectives. They don't all duplicate each other's notes. One focuses on fingerprints, another on alibis, another on motive. Once the case is solved, you could have done it with fewer detectives, but during the investigation, you needed the whole team so the right specialization could emerge naturally. Attention heads work the same way: during training, they each gravitate toward a different relationship type in the data. After training, many of them are doing similar enough work that you can drop a chunk without losing much quality.

Key concepts

Mechanistic interpretability Induction heads Head pruning Syntactic attention heads

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

Walk through Voita 2019 and Clark 2019 specialization findings, Olsson 2022 induction heads with their emergence pattern and ICL connection, Michel 2019 pruning evidence, and reconcile specialization with redundancy via the training vs inference distinction.

Real-world examples

Real products, models, and research that use this idea.

Voita et al. 2019 identified 'positional', 'syntactic', and 'rare token' head types in Transformer NMT encoders.
Clark et al. 2019 found BERT heads tracking dependency relations like direct-object and prepositional-object without explicit parse supervision.
Olsson et al. 2022 traced induction heads in GPT-2 and Anthropic models, showing they emerge during training and correlate with the onset of in context learning.
Michel et al. 2019 pruned 60-80% of attention heads from a translation transformer with negligible BLEU loss.
Anthropic's Transformer Circuits Thread has extended mechanistic interpretability to specific algorithmic primitives implemented by head combinations.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy do induction heads emerge sharply during training rather than gradually?

Sign in or go Pro to see the approach for this follow-up.

QHow does head pruning evidence interact with the multi-head structural argument?

Sign in or go Pro to see the approach for this follow-up.

QWhat experimental techniques do researchers use to identify head functions?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Asserting either that all heads are identical (wrong) or that no heads are redundant (also wrong). The interesting truth is both specialization and redundancy coexist.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.