Scaled Dot Product Attention

Name the scalar that QK^T is divided by inside scaled dot-product attention.

Attention temperature divides QK^T; sampling temperature divides output logits, distinguishwhere each lives.

Define a single attention head, what does one head actually own and produce?

Contrast ALiBi's linear distance bias with RoPE's vector rotation, which one modifies scores additively, and which one modifies Q and K directly?

Why is the dot product scaled by √d_k in attention? Derive the variance argument.

What specifically goes wrong if you remove the √d_k scaling from attention?

Complete the canonical attention formula: Attention(Q, K, V) = ___(QKᵀ / ___) · V