Fill in the scaled dot-product attention formula.

Attempt it

Attention(Q, K, V) = softmax(QKᵀ / ) ·

Quick reference

TL;DR

The first blank is the square root of d_k, the per-head key dimension. The second blank is V, the value matrix being weighted by the attention distribution.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine you have a question and a row of labelled boxes. You want to mix the contents of the boxes into one good answer. First, you compare your question card to every label and give each box a score. Big scores can shout over small ones, so you quiet them down by dividing by a fair amount tied to how detailed each label is. The square root part is just the right size of brake so no single label drowns out the rest. Now you have soft weights that say how much each box matters. But weights alone are not the answer. You still need to actually scoop the contents of each box and mix them by those weights. The contents are called V. The two missing pieces are the brake under the slash and the contents at the end.

Key concepts

Attention Mechanism

Detailed answer & concept explanation

~3 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Sign in for 5 free unlocks/day or go Pro for unlimited. See Pro pricing.

Quick scan · 2–4 min

Everything important, quickly.

Identify both blanks immediately, state the formula in full, then spend the rest of the time on why. Derive the variance argument that fixes the scale at the square root of d_k, explain why V is necessary at the end, and connect the formula to its production primitive.

Real-world examples

Real products, models, and research that use this idea.

The 'Attention Is All You Need' paper introduced exactly this formula with the same scaling factor that survives in every modern transformer.
PyTorch's `F.scaled_dot_product_attention` hardcodes `1/sqrt(head_dim)` as the scale, exposing it as an override only for niche use cases.
FlashAttention 2 and 3 reorder the computation to be memory-efficient but produce bit-equivalent outputs because the scale factor is part of the contract.
Hugging Face Transformers' Llama, Mistral, Qwen, and DeepSeek attention modules all apply the same `1/sqrt(head_dim)` factor in every layer.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is the constant `sqrt(d_k)` rather than `d_k` or `log(d_k)`?

Sign in or go Pro to see the approach for this follow-up.

QCould you absorb the scale factor into the Q or K projection weights?

Sign in or go Pro to see the approach for this follow-up.

QWhat changes in the formula for multi-head attention?

Sign in or go Pro to see the approach for this follow-up.

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Writing d_k instead of sqrt(d_k), or forgetting V altogether. The square root specifically cancels the standard deviation growth of dot products, and V is what the attention weights are blending.

Sign in or go Pro to see all red flags and common mistakes. See Pro pricing.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.