Name the scalar that QK^T is divided by inside scaled dot-product attention.

Flashcard·Easy·4.0 · 0·~30s

Attempt it

sqrt(d_k), where d_k is the per-head key dimension, same value as head_dim (= hidden_size / num_heads in standard configurations). Not sqrt(d_model) and not sqrt(num_heads). The division happens before softmax: scores = (QK^T) / sqrt(d_k). The reason is variance control: for d_k-dimensional unit-variance vectors, dot products have variance d_k, so without scaling the softmax inputs would saturate as d_k grows, collapsing the attention distribution to nearly one-hot and killing gradients on the other positions.

Your answer

Quick reference

TL;DR

sqrt(d_k), the per-head key dimension. Not sqrt(d_model) and not sqrt(num_heads). It keeps pre-softmax score variance near 1.

Memory aid

Easy to grasp

Picture a dart competition where each round you sum up scores from many dartboards at once. The more boards you add, the wider the range of total scores swings, just from random luck. To keep the leaderboard meaningful round to round, you divide the total by something that scales with how many boards you summed over, so a typical total stays around the same size. Attention does the same thing: the dot product between query and key is a sum over d_k terms, and dividing by the square root of d_k keeps the typical magnitude steady regardless of how big you make a single head.

Key concepts

Scaled dot product attention Per-head dimension d_k Softmax Variance scaling

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

State the divisor sqrt(d_k), derive it from the variance of a d_k-term dot product, contrast with wrong candidates (1, d_k, sqrt(d_model)), and explain why the scaling is invariant across head sizes.

Side-by-side

Candidate divisor	Effect on score variance	Softmax behavior
1 (no scaling)	Variance grows as d_k	Saturates at large d_k, near one-hot, gradient vanishes off-peak
sqrt(d_k)	Variance ≈ 1 regardless of d_k	Stable across head sizes; the canonical choice
d_k	Variance ≈ 1/d_k	Over-corrects; softmax never sharpens, attention is near-uniform
sqrt(d_model)	Variance ≈ d_k / d_model (wrong scale)	Wrong magnitude per head; under-corrects for multi-head

Real-world examples

Real products, models, and research that use this idea.

Vaswani et al. 2017, equation 1: explicitly motivates sqrt(d_k) via the variance argument in section 3.2.1.
Llama 4 Maverick, Llama 3 70B: d_k = 128, divisor = sqrt(128) ≈ 11.3.
Mistral 7B and Mixtral: d_k = 128 per attention head, same sqrt(128) scaling.
Gemma 4 27B: head_dim = 256, so the divisor is sqrt(256) = 16.
PyTorch's torch.nn.functional.scaled_dot_product_attention applies the sqrt(d_k) divisor automatically and dispatches to FlashAttention kernels under the hood.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is the scaling specifically sqrt(d_k) rather than d_k or 1/d_k?

QHow does FlashAttention apply the sqrt(d_k) scaling inside its blockwise softmax computation?

QWhy does RoPE preserve the sqrt(d_k) scaling, but int8 KV-cache quantization sometimes break it?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Writing sqrt(d_model) instead of sqrt(d_k). The scaling is per-head, so d_k = hidden_size / num_heads is the right value.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.