There are **three canonical dropout sites** in a transformer block:
1. **Attention dropout**: on the post-softmax attention weights, before they are used to weight the V vectors. This randomly zeroes some attention edges, preventing the model from over relying on specific token to token connections.
2. **FFN output dropout**: on the output of the feed-forward network, before it is added back onto the residual stream.
3. **Residual / post-sublayer dropout**: on the residual stream itself between blocks, after each sublayer has added its contribution.
**The two wrong placements** in the options:
- **Inside the norm**: RMSNorm and LayerNorm exist to stabilize activation statistics. Injecting random zeros inside the norm defeats the entire purpose.
- **On the Q, K, V projections**: dropout here would corrupt the dot-product distribution before the softmax, making attention noisy in unhelpful ways.
**Modern reality**: in large scale pretraining of frontier LLMs (Llama 3, GPT-4+), dropout is often disabled entirely because the model is data bound, not parameter bound. Instruction tuning and fine-tuning typically re-enable it at low rates (0.05 to 0.1).
**Three canonical sites:** attention weight dropout, FFN output dropout, and residual / post-sublayer dropout. These were defined in the original 2017 transformer paper and remain the standard placements.
**Why these three.**
- **Attention weights** are a soft index over tokens; dropping random entries forces the model not to over-rely on any single token to token edge.
- **FFN output** is where per token features are computed; dropping random dimensions encourages robust feature representations.
- **Residual stream** dropout regularizes the contribution of each sublayer to the running residual; this is the place that prevents over-reliance on any single block.
**Why NOT inside the norm.** Normalization computes per token statistics (mean and variance for LayerNorm, RMS for RMSNorm) and rescales to a target distribution. Dropout inside that computation would corrupt the statistics it relies on, breaking the variance stabilization the norm exists to provide. No published transformer implementation does this.
**Why NOT on Q/K/V projections.** The attention dot product `q · k` produces logits whose scale and distribution are calibrated by the `1 / sqrt(d_k)` scaling. Dropout on Q or K would inject random zeros directly into the dot product, distorting the logits unpredictably. Dropout on V would zero entire value vectors, which is too aggressive a perturbation for the soft attention mechanism. The model performs worse with these placements, which is why no production implementation uses them.
**Modern pretraining reality.** Llama 3.1, GPT-4 class, and Mistral pretraining runs typically set dropout to 0. The models are under-fit relative to the training data, so the regularizer just slows convergence. Instruction tuning and supervised fine-tuning often re-enable it at small rates (0.05 to 0.1) because those datasets are much smaller and overfitting is a real risk.
Quick reference
TL;DR
Three canonical dropout sites: attention weights (post-softmax), FFN output, and residual stream. Norm internals and Q/K/V projections do NOT get dropout.
Think of dropout like a coach who randomly benches a few players every practice so the team can't rely on one star to carry every game. In a transformer block, there are three sensible benches: right after the model decides which earlier words to focus on (so it doesn't get hooked on one specific word-to-word connection), on the output of the per-word processing layer (so each word learns sturdier features), and on the shared bus that carries information between blocks (so no single layer becomes load-bearing). Don't bench players inside the rescaling step itself (that ruins the whole point of stabilizing the signal), and don't bench the players that build the focus scores in the first place (that just adds useless noise). In modern frontier LLMs, dropout is often turned off during pretraining because the data is so vast the model is under-fit anyway.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Pretraining a frontier scale dense LLM on a trillion token dataset The model is data bound, not parameter bound. Dropout slows convergence without improving generalization; setting all rates to 0 is the modern default.
Implementing inference for an already trained model Dropout is a training time operation. At inference it's bypassed entirely (or replaced with its expected value equivalent in older deterministic inference setups).
Reaching for stochastic depth (DropPath) as a substitute DropPath drops entire sublayers and is a different regularizer. It can coexist with dropout but addresses a different problem (path length regularization rather than activation regularization).
Quick scan · 2–4 min
Everything important, quickly.
6 min: name the three canonical sites, explain the regularization role of each, justify why norm internals and Q/K/V projections are excluded, and discuss modern pretraining-vs-fine-tuning practice.
Real-world examples
Real products, models, and research that use this idea.
1BERT base (2018): dropout 0.1 on attention weights, FFN output, and residual stream: the canonical three.
2Llama 3.1 pretraining: dropout set to 0 across all sites; the model is data bound, not parameter bound.
3Mistral 7B instruction tuning: dropout 0.05 on FFN output and residual stream re-enabled to combat overfitting on small SFT datasets.
4Hugging Face's `LlamaConfig.attention_dropout` defaults to 0.0 but exposes the parameter for fine-tuning use cases.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is attention dropout applied to post-softmax weights rather than pre-softmax logits?
Sign in or go Pro to see the approach for this follow-up.
QWhy does frontier pretraining set dropout to 0 when classic ML wisdom says dropout always helps generalization?
Sign in or go Pro to see the approach for this follow-up.
QHow does dropout interact with FlashAttention's fused kernel?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Placing dropout inside the RMSNorm computation. The norm exists to stabilize activation statistics; injecting dropout there defeats its purpose. Same with Q/K/V projections; dropout there destabilizes the attention dot product.