Head-pruning studies report that many attention heads can be removed with negligible loss. Pick the correct interpretation.

**Head pruning experiments consistently show that a sizable fraction of attention heads in a trained transformer can be removed at inference time with only a small quality drop.** Michel et al. (2019) and several follow-ups demonstrated this on BERT and GPT-class models. The natural conclusion 'so multi-head attention is wasteful' is wrong. The correct interpretation requires separating two questions: - **Can you prune heads from a trained model?** Empirically, yes. Many heads have small contribution to downstream loss, and removing them produces a model that runs faster with negligible quality loss. - **Should you train with fewer heads from scratch?** Empirically, no. Training a model with the pruned head count (rather than pruning after) generally produces worse final quality. The redundant heads matter during training even if they become slack at inference. The mechanism is **over-provisioning**. During gradient-based training, more heads than strictly needed give the optimizer parallel hypotheses, redundancy that smooths the loss landscape, and exploration headroom. Once training converges, the surviving useful heads carry most of the signal, and the rest can be pruned. The other two MCQ options are wrong: heads are not linearly dependent at initialization (they have independent random init); and head-pruning results replicate across evaluation sets, so they are not benchmark artifacts.

Head pruning is one of the cleanest empirical findings in transformer interpretability: **many attention heads can be removed post-training with small loss, but you cannot reach the same final quality by training with fewer heads from the start.** **The empirical pattern.** Michel et al. (2019) and subsequent work showed that for BERT, GPT-2, and Transformer-based NMT models, 30-50% of attention heads can be pruned with under 1 BLEU or perplexity point of loss. The result replicates across model families and tasks. **The correct interpretation is over-provisioning.** Multi-head attention provides redundancy during training that helps optimization. More parallel hypotheses smooth the loss landscape and let the model explore different attention specializations in parallel. Once training converges, the model has learned which heads carry meaningful signal, and the redundant ones can be removed. **Two nuances matter.** First, prunable heads are identified empirically after training. You measure each head's contribution to downstream loss (via gradient-based importance scoring or ablation), rank, and prune the bottom k. Second, train time equivalent reductions hurt quality. A model trained with num_heads = 16 from scratch hits worse final perplexity than a 32-head model pruned to 16. The extra heads do real work during training even if they become redundant after. **The practical takeaway.** Head pruning is a viable inference-time optimization when you need to shrink memory or latency, but it is not an architectural recommendation to use fewer heads. Modern LLM design still uses 32-64+ heads at frontier scale because that count produces the best final model.