Identify the locations where dropout typically sits inside a transformer block

Multi-select·Medium·4.0 · 0·~1 min·Asked atNetflix NVIDIA·Relevant atMistral AI

Attempt it

There are **three standard dropout sites** in a transformer block (dropout randomly zeroes values during training): 1. **Attention dropout**: on the post-softmax attention weights, before they are used to weight the V vectors. This randomly zeroes some attention edges, preventing the model from over relying on specific token to token connections. 2. **FFN output dropout**: on the output of the feed-forward network, before it is added back onto the residual stream. 3. **Residual / post-sublayer dropout**: on the residual stream itself between blocks, after each sublayer has added its contribution. **The two wrong placements** in the options: - **Inside the norm**: RMSNorm and LayerNorm exist to stabilize activation statistics. Injecting random zeros inside the norm defeats the entire purpose. - **On the Q, K, V projections**: dropout here would corrupt the dot-product distribution before the softmax, making attention noisy in unhelpful ways. **Modern reality**: in large scale pretraining of frontier LLMs (Llama 3, GPT-4+), dropout is often disabled entirely. The reason is that the model is data bound, not parameter bound. Instruction tuning and fine-tuning typically re-enable it at low rates (0.05 to 0.1).

**Three standard sites:** attention weight dropout, FFN output dropout, and residual / post-sublayer dropout. These were defined in the original 2017 transformer paper and remain the standard placements. **Why these three.** - **Attention weights** are a soft index over tokens. Dropping random entries forces the model not to over-rely on any single token to token edge. - **FFN output** is where per token features are computed. Dropping random dimensions encourages robust feature representations. - **Residual stream** dropout regularizes the contribution of each sublayer to the running residual. **Why NOT inside the norm.** Normalization computes per token statistics (mean and variance for LayerNorm, RMS for RMSNorm) and rescales to a target distribution. Dropout inside that computation would corrupt the statistics it relies on, breaking the variance stabilization the norm exists to provide. **Why NOT on Q/K/V projections.** The attention dot product `q · k` produces logits whose scale and distribution are calibrated by the `1 / sqrt(d_k)` scaling. Dropout on Q or K would inject random zeros directly into the dot product, distorting the logits unpredictably. Dropout on V would zero entire value vectors, which is too aggressive a perturbation for the soft attention mechanism. The model performs worse with these placements. **Modern pretraining reality.** Llama 3.1 and Mistral pretraining runs typically set dropout to 0. The models are underfit relative to the training data, so the regularizer only slows convergence. Instruction tuning and fine-tuning often re-enable it at small rates (0.05 to 0.1) because those datasets are much smaller.

TL;DR

Three canonical dropout sites: attention weights (post-softmax), FFN output, and residual stream. Norm internals and Q/K/V projections do NOT get dropout.

Memory aid

Easy to grasp

Think of dropout like a coach who randomly benches a few players every practice so the team can't rely on one star. In a transformer block, there are three sensible benches. The first is right after the model decides which earlier words to focus on (so it doesn't get hooked on one word-to-word connection). The second is on the output of the per-word processing layer (so each word learns sturdier features). The third is on the shared bus that carries information between blocks (so the model does not depend on any single layer). Don't bench players inside the rescaling step (that ruins the whole point of stabilizing the signal). Don't bench the players that build the focus scores either (that adds useless noise). In frontier LLMs, dropout is often turned off during pretraining because the data is so vast the model is under-fit anyway.

Key concepts

Transformer architecture Attention

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Dropout placement inside a transformer block is one of those details that seems trivial until you try to put it in the wrong place. The 2017 paper specified three standard sites (attention weights, FFN output, residual stream) and almost no published architecture has deviated. The placements are not arbitrary. They reflect a careful choice about which activation surfaces are well conditioned for stochastic regularization and which would be destabilized by it.

The three standard sites

Attention dropout sits on the post-softmax attention weights, before the weighted sum over V. The softmax produces a distribution over tokens for each query. Dropout zeros random entries in that distribution. Implementations vary on whether the result is renormalized, but the standard is to drop without renormalization, accepting the lossy sum. The regularization effect is to perturb which tokens get attended to on each forward pass. This prevents the model from over-relying on any single token to token edge.

FFN output dropout sits on the output of the position-wise feed-forward network, before the residual add. This is standard MLP dropout: zero random dimensions of the FFN output vector, forcing the model to maintain redundant feature representations.

Residual / post-sublayer dropout sits on the sublayer output before it is added to the residual stream. This regularizes how much each sublayer contributes to the running residual. It's the strongest dropout location because it perturbs the residual stream itself, which every later sublayer reads from.

In BERT, T5, and the original 2017 transformer, these three sites all use the same dropout rate (typically 0.1). Modern setups sometimes vary the rate per site or disable some sites entirely.

Why norm internals are off-limits

Why Q/K/V projection dropout is destructive

Modern pretraining reality: dropout often off

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

BERT base (2018): dropout 0.1 on attention weights, FFN output, and residual stream: the canonical three.
Llama 3.1 pretraining: dropout set to 0 across all sites. The model is data bound, not parameter bound.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is attention dropout applied to post-softmax weights rather than pre-softmax logits?

Pre-softmax dropout would zero some logits. After softmax, the remaining nonzero logits would absorb almost all the probability mass via the exp normalization, defeating the regularization. Post-softmax dropout zeros some attention edges while keeping the rest correctly normalized (after renormalization).

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Placing dropout inside the RMSNorm computation. The norm exists to stabilize activation statistics. Injecting dropout there defeats its purpose. Dropout on Q/K/V projections is also wrong because it destabilizes the attention dot product.

60 second bullets to scan on the way to the call.

The three canonical dropout sites in a transformer block
Why post-softmax weights are the right attention dropout target, not Q/K/V projections

Primary sources. Browse if you want the original framing.