Walk through how a token's d_model embedding is split into per-head pieces inside multi-head attention.

Multi-head attention splits the work across `num_heads` parallel heads. The split looks more complicated than it is, it is really just a reshape. Start from a token embedding `x` of shape `(d_model,)`. Three projections produce Q, K, and V: 1. `Q = x @ W_Q` → shape `(d_model,)` 2. `K = x @ W_K` → shape `(d_model,)` 3. `V = x @ W_V` → shape `(d_model,)` Now Q, K, V are still d_model wide. The 'split into heads' step is a **view + transpose**: - Reshape Q from `(d_model,)` to `(num_heads, d_head)` where `d_head = d_model / num_heads`. - No memory copy; PyTorch reinterprets contiguous chunks of the same buffer as a new axis. Each head then performs scaled dot-product attention on its own `(d_head,)`-wide slice. At the end, the per-head outputs are concatenated back into a `(d_model,)` vector and run through the output projection `W_O`. So the answer to 'how is the embedding split' is: through one big projection followed by a reshape that re-labels contiguous chunks as heads.

Multi-head attention does **not** duplicate the embedding per head. It uses a single batched projection followed by a reshape. Given input `x` of shape `(batch, seq, d_model)`, the pipeline is: 1. `Q = x @ W_Q`, same for K and V. Each `W_*` is `(d_model, d_model)`. Output stays `(batch, seq, d_model)`. 2. Reshape Q, K, V to `(batch, seq, num_heads, d_head)` where `d_head = d_model / num_heads`. This is a **view**, not a copy. 3. Transpose to `(batch, num_heads, seq, d_head)` so each head's tensor is contiguous along the sequence axis. 4. Run scaled dot-product attention independently per head: `softmax(Q_h K_h^T / sqrt(d_head)) V_h` for each head h. 5. Concatenate the heads back along the last axis into `(batch, seq, d_model)`. 6. Apply the output projection `W_O` of shape `(d_model, d_model)`. The critical insight: the d_model-wide Q projection IS the per-head Q projections concatenated. The model learns one big matrix; the reshape just exposes the head structure. There is no copy, no duplication, no extra parameters compared to a hypothetical 'one projection per head' design.