Multi Head Attention

Given Q of shape (B, n_heads, T, d_head), the per-head attention output before concatenation has shape ___.

Describe the W_O projection in multi-head attention, its shape and what it mixes.

Beyond shape preservation, name two things W_O does after head concat

Spot the GQA configuration error: num_attention_heads=32, num_key_value_heads=7.

Walk through how a token's d_model embedding is split into per-head pieces inside multi-head attention.

Define a single attention head, what does one head actually own and produce?

Why does multi-head attention outperform a single large attention head?

What does multi-head attention give you that a single attention head with the same total parameters cannot?

True or false: switching from 1-head to 12-head attention at the same d_model increases total parameter count.

If d_model = 768 and num_heads = 12, what is d_head?