True or false: swapping a single-head attention layer for a 12-head one (same d_model) raises the parameter count.

**False.** Swapping a single-head attention layer for a 12-head one with the same d_model does NOT change the parameter count. The Q, K, V projection matrices are `d_model x d_model` in both cases. In single-head attention, you have one Q, one K, one V projection, each a `d_model x d_model` matrix. Total parameters in the projection layers: `4 x d_model^2` (counting the output projection W_O). In multi-head attention with h heads, you have... the same matrices. They are just viewed differently. A `d_model x d_model` projection can be reshaped into `d_model x h x d_head` where `d_head = d_model / h`. The reshape is a view, not new parameters. What changes is HOW you use those parameters at compute time. The single big matrix is split into h smaller blocks. Each block runs its own scaled dot-product attention in parallel. The outputs are concatenated back together and passed through W_O. The whole point of multi-head attention is parallel specialization at the same parameter cost. You do not pay extra in weights; you gain expressiveness by letting h heads attend to different relations inside the same parameter budget.

**False.** The Q, K, V, and O projections are each `d_model x d_model` regardless of head count. Multi-head attention is a reshape of these big matrices into `h` smaller blocks of width `d_head = d_model / h`. Specifically, for a single attention layer: - W_Q: shape `(d_model, d_model)`, viewed as `(d_model, h, d_head)`. - W_K: shape `(d_model, d_model)`, viewed as `(d_model, h, d_head)`. - W_V: shape `(d_model, d_model)`, viewed as `(d_model, h, d_head)`. - W_O: shape `(d_model, d_model)`, viewed as `(h * d_head, d_model)`. Total parameter count: `4 * d_model^2`, independent of h. Going from h=1 to h=12 changes nothing in the weight count. What the multi-head structure buys is parallel specialization. Each head sees a smaller `d_head`-wide subspace and learns to attend to a specific kind of relation: positional adjacency, syntactic structure, coreference, induction. Probing studies on BERT and GPT have repeatedly found this specialization without any explicit constraint at training time. The practical insight: if you want more attention capacity, increase d_model (which quadratically grows the projection parameters) or add more layers. Splitting an existing d_model across more heads only reshapes the existing budget, it does not add capacity.