Head pruning studies report that many attention heads can be removed with negligible loss. Pick the correct interpretation.

MCQ·Medium·4.0 · 0·~1 min·Asked atEy Mphasis Patronus·Relevant atMeta

Attempt it

**Head pruning experiments consistently show that a sizable fraction of attention heads in a trained transformer can be removed at inference time with only a small quality drop.** Michel et al. (2019) and several follow-ups demonstrated this on BERT and GPT-class models. The natural conclusion 'so multi-head attention is wasteful' is wrong. The correct interpretation requires separating two questions: - **Can you prune heads from a trained model?** Empirically, yes. Many heads have small contribution to downstream loss, and removing them produces a model that runs faster with negligible quality loss. - **Should you train with fewer heads from scratch?** Empirically, no. Training a model with the pruned head count (rather than pruning after) generally produces worse final quality. The redundant heads matter during training even if they become slack at inference. The mechanism is **over-provisioning**. During gradient-based training, more heads than strictly needed give the optimizer parallel hypotheses, redundancy that smooths the loss landscape, and exploration headroom. Once training converges, the surviving useful heads carry most of the signal, and the rest can be pruned. The other two MCQ options are wrong: heads are not linearly dependent at initialization (they have independent random init); and head-pruning results replicate across evaluation sets, so they are not benchmark artifacts.

Head pruning is one of the cleanest empirical findings in transformer interpretability: **many attention heads can be removed post-training with small loss, but you cannot reach the same final quality by training with fewer heads from the start.** **The empirical pattern.** Michel et al. (2019) and subsequent work showed that for BERT, GPT-2, and Transformer-based NMT models, 30-50% of attention heads can be pruned with under 1 BLEU or perplexity point of loss. The result replicates across model families and tasks. **The correct interpretation is over-provisioning.** Multi-head attention provides redundancy during training that helps optimization. More parallel hypotheses smooth the loss landscape and let the model explore different attention specializations in parallel. Once training converges, the model has learned which heads carry meaningful signal, and the redundant ones can be removed. **Two nuances matter.** First, prunable heads are identified empirically after training. You measure each head's contribution to downstream loss (via gradient-based importance scoring or ablation), rank, and prune the bottom k. Second, train time equivalent reductions hurt quality. A model trained with num_heads = 16 from scratch hits worse final perplexity than a 32-head model pruned to 16. The extra heads do real work during training even if they become redundant after. **The practical takeaway.** Head pruning is a viable inference-time optimization when you need to shrink memory or latency, but it is not an architectural recommendation to use fewer heads. Modern LLM design still uses 32-64+ heads at frontier scale because that count produces the best final model.

TL;DR

Pretraining over-provisions heads. Many can be pruned post-hoc with small quality loss, but training with fewer heads from scratch hits worse final quality. The extra heads pay off at train time.

Memory aid

Easy to grasp

Picture a relay team training with 16 runners. After many races, you notice only 8 actually run the final routes; the other 8 are bench helpers, warming up, scouting, sharing tactics. Once the team is trained, you can race with just the 8 runners and barely lose. But if you had only ever signed 8 runners, the team never would have developed the same tactics, because the bench helpers were what made the training useful. Attention heads work the same way: the extras help while learning, and you can drop them after.

Key concepts

Head pruning Multi-head attention Training-time over-provisioning Attention interpretability

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

The head-pruning literature is one of the cleanest examples in transformer research of how empirical findings can mislead if interpreted naively. Many attention heads in a trained transformer can be removed with negligible quality loss, this is robustly true. But the natural conclusion 'so multi-head attention is wasteful' is empirically wrong: training with fewer heads from scratch produces worse final quality.

The right interpretation is over-provisioning. Extra heads provide scaffolding during training that helps the optimizer reach a better solution, even if that scaffolding becomes redundant at inference time. This deep dive walks the canonical empirical results, the over-provisioning mechanism, how it ties into interpretability findings about head specialization, why modern LLM design still uses large head counts despite the prunability result, and how GQA and MLA represent structured ways to capture the head-redundancy insight without sacrificing train-time quality.

The empirical landscape

Michel et al. 2019 (Are Sixteen Heads Really Better Than One?) was the canonical demonstration. The paper showed that on BERT for the GLUE benchmark and on a Transformer-based machine translation model:

30-50% of attention heads could be removed (via greedy importance-ranking pruning) with under 1 metric point of quality loss.
Some individual layers could be reduced to a single head with negligible loss.
The remaining 'critical' heads varied across layers but tended to cluster around specific syntactic or semantic roles.

How prunable heads are identified

The method is empirical: rank heads by a gradient-based importance score (proportional to how much loss increases when the head is masked off), prune the lowest-ranked, retrain or fine-tune briefly to recover quality. The result is a smaller model with similar evaluation metrics.

Variants of the method include:

Magnitude-based pruning: rank by L2 norm of the head's output projection.
Gradient-based: rank by the head's contribution to the training loss gradient.
Ablation-based: rank by the increase in loss when the head is zeroed.

All three give roughly similar rankings, and all three identify a substantial prunable fraction in BERT and GPT-class models.

Replication across model families

Follow-up work replicated the finding on:

T5 (Raffel et al. 2020) and its variants.
GPT-2 and GPT-3 (limited public studies, but similar patterns reported).
Vision Transformers (with somewhat smaller prunable fractions).
BART, Pegasus, and other encoder-decoder models.

The prunable fraction varies by model and task, but the qualitative pattern, many heads are dispensable post-training, is robust.

The empirical result is solid. The question is what to do with it, and the naive answer 'train with fewer heads' is the trap.

Why over-provisioning is the right interpretation

Head specialization and the interpretability angle

Why modern LLM design still uses large head counts

The senior-level frame

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Michel et al. 2019 'Are Sixteen Heads Really Better Than One?' showed 30-50% of heads prunable on BERT and Transformer NMT.
Voita et al. 2019 connected head pruning to attention specialization, identifying syntactic, positional, and topic head patterns.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does training with fewer heads from scratch produce worse quality than pruning after?

Gradient-based optimization is path-dependent. With more heads, the optimizer has parallel hypotheses, redundancy that smooths the loss landscape, and exploration headroom. With fewer heads, each gradient step has less freedom and the optimizer is more likely to get stuck in worse local minima. The extra heads are scaffolding for the training process; once training converges, the scaffolding can be removed but it had to be there during the build.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Concluding 'pruning works, so train with fewer heads.' Empirically that fails: smaller from scratch models hit worse quality. The redundancy is a training-time enabler, not a wasted capacity.

60 second bullets to scan on the way to the call.

Michel et al. 2019 result: 30-50% of heads prunable post-training with small loss
Why training with fewer heads from scratch produces worse final quality

Primary sources. Browse if you want the original framing.