Decode the acronym MHA in the transformer attention context.

Flashcard·Easy·4.0 · 0·~30s

Attempt it

MHA stands for Multi-Head Attention, the baseline attention pattern from Vaswani 2017 where every attention layer runs h independent (Q, K, V) projection sets in parallel, computes scaled dot-product attention per head, concatenates the per-head context vectors back into a d_model-wide vector, and finishes with the output projection W_O. Each head has its own K and its own V (no sharing across heads). This is the configuration MQA and GQA are KV-cache-saving variants of: MQA collapses to one shared K, V across all query heads; GQA shares K, V within groups of query heads.

Your answer

Quick reference

TL;DR

MHA stands for Multi-Head Attention, the Vaswani 2017 baseline where h independent (Q, K, V) heads run in parallel inside one attention layer.

Memory aid

Easy to grasp

Picture a panel of expert reviewers reading the same document. Each reviewer has a different angle: one tracks grammar, one tracks logic, one tracks tone, one tracks references. They all read the same text at the same time, write their notes independently, and the editor then merges the notes into a final report. MHA is that panel inside one step of a transformer. Several reviewers (called heads) work on the same sentence at the same time, each with their own private set of reading glasses, each picking up on a different kind of relationship between words. At the end, their notes are stitched together and combined into one summary the model uses to keep thinking.

Key concepts

Multi-head attention Transformer architecture

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

3m: what MHA stands for, the parallel-head mechanism, why parameters are unchanged vs single-head, parallel specialization across heads, and the relationship to MQA, GQA, MLA.

Real-world examples

Real products, models, and research that use this idea.

The original Transformer in Vaswani et al. 2017 used MHA with h=8 heads at d_model=512.
BERT-base used MHA with h=12 heads at d_model=768; BERT-large used h=16 at d_model=1024.
GPT-2 and GPT-3 used MHA throughout, before the field shifted to GQA in larger production decoders.
Llama 4 Maverick and Mistral Large 3 use GQA, the production successor to MHA that keeps multi-head queries but shares K and V.
DeepSeek V4 uses MLA, another MHA descendant that compresses K and V into a low rank latent while preserving multi-head queries.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does the output projection W_O matter rather than just concatenating heads?

QIf MHA has the same parameter count as single-head, why is the multi-head version empirically better?

QHow would you describe the relationship between MHA and GQA in one sentence?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Reading MHA as 'masked head attention' or thinking the h heads increase parameter count. Heads partition the existing d_model budget; the M stands for Multi, not Masked.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.