Pick the formula that defines head_dim from hidden_size and num_heads.

**The formula is `head_dim = hidden_size / num_heads`**, with the requirement that `hidden_size` is exactly divisible by `num_heads`. The reason is mechanical. Multi-head attention takes the per-token hidden vector of size `hidden_size` and reshapes it into a 2D block of shape `(num_heads, head_dim)` so each head can compute attention independently on its slice. The reshape requires the two output dimensions to multiply back to the input size: ``` hidden_size = num_heads * head_dim ``` Rearranging gives `head_dim = hidden_size / num_heads`. The integer divide is implicit because PyTorch's `view` and `reshape` operations require the dimensions to match exactly, no rounding allowed. Worked examples: - GPT-2 small: hidden_size = 768, num_heads = 12, so head_dim = 64. - Llama-2 7B: hidden_size = 4096, num_heads = 32, so head_dim = 128. - Llama-2 70B: hidden_size = 8192, num_heads = 64, so head_dim = 128. The other answer choices are wrong: - Multiplying instead of dividing would give an absurd value (768 * 12 = 9216). - Taking sqrt would produce a non-integer in nearly every case. - Treating head_dim as free contradicts how the reshape inside attention actually works.

**head_dim = hidden_size / num_heads**, integer divide, with `hidden_size % num_heads == 0` required. The relationship is fixed by the multi-head attention reshape. The Q, K, V projections produce tensors of shape `(batch, seq_len, hidden_size)`. To run attention per head, the implementation views these as `(batch, seq_len, num_heads, head_dim)`, then transposes to `(batch, num_heads, seq_len, head_dim)` for the matmul. The view operation requires the trailing dimensions to factor back to hidden_size exactly. The identity: $$d_\text{head} = \frac{d_\text{model}}{n_\text{heads}}$$ Applied to standard configurations: - BERT base / GPT-2 small: 768 / 12 = 64. - Llama-2 7B / Mistral 7B: 4096 / 32 = 128. - Llama-2 70B: 8192 / 64 = 128. The divisibility constraint is enforced at model construction. PyTorch's `MultiheadAttention` and Hugging Face's attention implementations both raise an error if `hidden_size % num_heads != 0`. **Why this matters for design choices.** You cannot pick hidden_size, num_heads, and head_dim independently. Most modern LLMs fix head_dim at 128 (kernel-friendly) and pick hidden_size + num_heads to satisfy the constraint. A few research models (some MLA variants, some MoE configs) add an explicit projection layer between the linear and the heads, which lets head_dim be set independently. The default and the interview-correct answer is the simple integer division.