Multi Head

Given Q of shape (B, n_heads, T, d_head), the per head attention output before concatenation has shape ___.

Describe the W_O projection in multi-head attention, its shape and what it mixes.

Beyond shape preservation, name two things W_O does after head concat

Which config token names the count of parallel attention heads in a layer?

PremiumTrue or false: swapping…

Decode the acronym MHA in the transformer attention context.

Across which of these axes is the KV cache stored: per layer, per head, per batch, or all three?

Head pruning studies report that many attention heads can be removed with negligible loss. Which is the correct interpretation?

Which formula defines head_dim from hidden_size and num_heads?

What are the typical head counts and head dimensions for modern 7B models?

Walk through how a token's d_model embedding is split into per head pieces inside multi-head attention.

What is in a single attention head, and what is its output?

Why does multi-head attention outperform a single large attention head?

PremiumWhat does multi-head attention…

True or false: switching from 1-head to 12-head attention at the same d_model increases total parameter count.