**Head pruning experiments consistently show that a sizable fraction of attention heads in a trained transformer can be removed at inference time with only a small quality drop.** Michel et al. (2019) and several follow-ups demonstrated this on BERT and GPT-class models. The natural conclusion 'so multi-head attention is wasteful' is wrong.
The correct interpretation requires separating two questions:
- **Can you prune heads from a trained model?** Empirically, yes. Many heads have small contribution to downstream loss, and removing them produces a model that runs faster with negligible quality loss.
- **Should you train with fewer heads from scratch?** Empirically, no. Training a model with the pruned head count (rather than pruning after) generally produces worse final quality. The redundant heads matter during training even if they become slack at inference.
The mechanism is **over-provisioning**. During gradient-based training, more heads than strictly needed give the optimizer parallel hypotheses, redundancy that smooths the loss landscape, and exploration headroom. Once training converges, the surviving useful heads carry most of the signal, and the rest can be pruned.
The other two MCQ options are wrong: heads are not linearly dependent at initialization (they have independent random init); and head-pruning results replicate across evaluation sets, so they are not benchmark artifacts.
Head pruning is one of the cleanest empirical findings in transformer interpretability: **many attention heads can be removed post-training with small loss, but you cannot reach the same final quality by training with fewer heads from the start.**
**The empirical pattern.** Michel et al. (2019) and subsequent work showed that for BERT, GPT-2, and Transformer-based NMT models, 30-50% of attention heads can be pruned with under 1 BLEU or perplexity point of loss. The result replicates across model families and tasks.
**The correct interpretation is over-provisioning.** Multi-head attention provides redundancy during training that helps optimization. More parallel hypotheses smooth the loss landscape and let the model explore different attention specializations in parallel. Once training converges, the model has learned which heads carry meaningful signal, and the redundant ones can be removed.
**Two nuances matter.**
First, prunable heads are identified empirically after training. You measure each head's contribution to downstream loss (via gradient-based importance scoring or ablation), rank, and prune the bottom k.
Second, train time equivalent reductions hurt quality. A model trained with num_heads = 16 from scratch hits worse final perplexity than a 32-head model pruned to 16. The extra heads do real work during training even if they become redundant after.
**The practical takeaway.** Head pruning is a viable inference-time optimization when you need to shrink memory or latency, but it is not an architectural recommendation to use fewer heads. Modern LLM design still uses 32-64+ heads at frontier scale because that count produces the best final model.
Quick reference
TL;DR
Pretraining over-provisions heads. Many can be pruned post-hoc with small quality loss, but training with fewer heads from scratch hits worse final quality. The extra heads pay off at train time.
Picture a relay team training with 16 runners. After many races, you notice only 8 actually run the final routes; the other 8 are bench helpers, warming up, scouting, sharing tactics. Once the team is trained, you can race with just the 8 runners and barely lose. But if you had only ever signed 8 runners, the team never would have developed the same tactics, because the bench helpers were what made the training useful. Attention heads work the same way: the extras help while learning, and you can drop them after.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Deciding initial num_heads for a model you plan to train from scratch Head-pruning results do not imply you should start with fewer heads; over-provisioning at train time is what makes the model trainable to high quality.
Architectures with GQA already applied GQA captures a different form of head redundancy structurally; layering post-hoc head pruning on top of GQA produces diminishing returns and adds complexity.
Models trained with curriculum learning or staged training Specialized training regimes may already exploit head redundancy in different ways; the standard prunability fraction may not apply.
Very small models where every head is already specialized Below a certain model size, all attention heads carry distinct signal and the prunability fraction shrinks toward zero; the result is a property of over-provisioned models, not all multi-head designs.
Quick scan · 2–4 min
Everything important, quickly.
60s: state the finding (30-50% of heads prunable post-training), give the over-provisioning interpretation, note the train from scratch failure as the key nuance, mention that prunable heads are identified empirically, connect to GQA / MLA as structured forms of the same insight.
Real-world examples
Real products, models, and research that use this idea.
1Michel et al. 2019 'Are Sixteen Heads Really Better Than One?' showed 30-50% of heads prunable on BERT and Transformer NMT.
2Voita et al. 2019 connected head pruning to attention specialization, identifying syntactic, positional, and topic head patterns.
3GQA in Llama-2 70B is a structured form of the same insight applied to the KV side: many K, V heads were redundant.
4MLA in DeepSeek V4 takes the head-redundancy lesson further with learned-latent compression of K, V across heads.
5Modern LLM design (Llama 4 Maverick, Mistral Large 3, Qwen 3.5, Gemma 4) still uses 32-64+ heads despite the prunability finding, because training quality dominates inference slack.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does training with fewer heads from scratch produce worse quality than pruning after?
Sign in or go Pro to see the approach for this follow-up.
QHow does the over-provisioning argument relate to GQA's KV head reduction?
Sign in or go Pro to see the approach for this follow-up.
QAre there cases where train from scratch with fewer heads does match the post-pruned model?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Concluding 'pruning works, so train with fewer heads.' Empirically that fails: smaller from scratch models hit worse quality. The redundancy is a training-time enabler, not a wasted capacity.