Multi Head

Given Q of shape (B, n_heads, T, d_head), the per-head attention output before concatenation has shape ___.

Describe the W_O projection in multi-head attention, its shape and what it mixes.

Beyond shape preservation, name two things W_O does after head concat

Which config token names the count of parallel attention heads in a layer?

True or false: swapping a single-head attention layer for a 12-head one (same d_model) raises the parameter count.

Decode the acronym MHA in the transformer attention context.

Across which axes is the KV cache stored, per layer, per head, per batch?

Head-pruning studies report that many attention heads can be removed with negligible loss. Pick the correct interpretation.

Pick the formula that defines head_dim from hidden_size and num_heads.

Modern 7B-class LLMs, fill in typical head counts and the resulting per-head dimension

Walk through how a token's d_model embedding is split into per-head pieces inside multi-head attention.

Define a single attention head, what does one head actually own and produce?

Why does multi-head attention outperform a single large attention head?

What does multi-head attention give you that a single attention head with the same total parameters cannot?

True or false: switching from 1-head to 12-head attention at the same d_model increases total parameter count.