Identify the locations where dropout typically sits inside a transformer block

There are **three canonical dropout sites** in a transformer block: 1. **Attention dropout**: on the post-softmax attention weights, before they are used to weight the V vectors. This randomly zeroes some attention edges, preventing the model from over relying on specific token to token connections. 2. **FFN output dropout**: on the output of the feed-forward network, before it is added back onto the residual stream. 3. **Residual / post-sublayer dropout**: on the residual stream itself between blocks, after each sublayer has added its contribution. **The two wrong placements** in the options: - **Inside the norm**: RMSNorm and LayerNorm exist to stabilize activation statistics. Injecting random zeros inside the norm defeats the entire purpose. - **On the Q, K, V projections**: dropout here would corrupt the dot-product distribution before the softmax, making attention noisy in unhelpful ways. **Modern reality**: in large scale pretraining of frontier LLMs (Llama 3, GPT-4+), dropout is often disabled entirely because the model is data bound, not parameter bound. Instruction tuning and fine-tuning typically re-enable it at low rates (0.05 to 0.1).

**Three canonical sites:** attention weight dropout, FFN output dropout, and residual / post-sublayer dropout. These were defined in the original 2017 transformer paper and remain the standard placements. **Why these three.** - **Attention weights** are a soft index over tokens; dropping random entries forces the model not to over-rely on any single token to token edge. - **FFN output** is where per token features are computed; dropping random dimensions encourages robust feature representations. - **Residual stream** dropout regularizes the contribution of each sublayer to the running residual; this is the place that prevents over-reliance on any single block. **Why NOT inside the norm.** Normalization computes per token statistics (mean and variance for LayerNorm, RMS for RMSNorm) and rescales to a target distribution. Dropout inside that computation would corrupt the statistics it relies on, breaking the variance stabilization the norm exists to provide. No published transformer implementation does this. **Why NOT on Q/K/V projections.** The attention dot product `q · k` produces logits whose scale and distribution are calibrated by the `1 / sqrt(d_k)` scaling. Dropout on Q or K would inject random zeros directly into the dot product, distorting the logits unpredictably. Dropout on V would zero entire value vectors, which is too aggressive a perturbation for the soft attention mechanism. The model performs worse with these placements, which is why no production implementation uses them. **Modern pretraining reality.** Llama 3.1, GPT-4 class, and Mistral pretraining runs typically set dropout to 0. The models are under-fit relative to the training data, so the regularizer just slows convergence. Instruction tuning and supervised fine-tuning often re-enable it at small rates (0.05 to 0.1) because those datasets are much smaller and overfitting is a real risk.