Multi Head

Given Q of shape (B, n_heads, T, d_head), the per head attention output before concatenation has shape ___.

Describe the W_O projection in multi-head attention, its shape and what it mixes.

Beyond shape preservation, name two things W_O does after head concat

Which config token names the count of parallel attention heads in a layer?

PremiumTrue or false: swapping…

Decode the acronym MHA in the transformer attention context.

Across which axes is the KV cache stored, per layer, per head, per batch?

Head pruning studies report that many attention heads can be removed with negligible loss. Pick the correct interpretation.

Pick the formula that defines head_dim from hidden_size and num_heads.

Modern 7B class LLMs, fill in typical head counts and the resulting per head dimension

Walk through how a token's d_model embedding is split into per head pieces inside multi-head attention.

Define a single attention head, what does one head actually own and produce?

Why does multi-head attention outperform a single large attention head?

PremiumWhat does multi-head attention…

True or false: switching from 1-head to 12-head attention at the same d_model increases total parameter count.