Softmax

PremiumOrder the tail end…

Does asking for top-k logprobs add real compute cost per token, or only bytes on the wire?

How does the temperature parameter reshape the sampling distribution?

At which step in the attention pipeline does the mask take effect?

Each row of the post-softmax attention weight matrix corresponds to which side of the QK product?

What does dividing pre-softmax attention scores by an extra factor > 1 do at inference?

What happens to attention weights when pre-softmax scores are divided by a temperature T > 1?

Along which axis of the QK^T score matrix is softmax applied inside attention?

Name the two properties softmax guarantees for every row of the attention weight matrix

Which token most often serves as an attention sink in pretrained autoregressive LLMs?

Why not just retrain models to eliminate attention sinks entirely?

How does softmax turn an attention score into an attention weight?

Name the scalar that QK^T is divided by inside scaled dot product attention.

Find the error in this claim about softmax over a fully masked attention row.

Where inside the attention sub-layer is dropout typically applied during training, and what does dropping there teach the model?

What does the dot product q . k measure inside an attention layer?

Identify the context vector in scaled dot product attention and what produces it.

What value does a 0-indexed causal mask store at row 2, column 3 for a 4-token sequence?

If attention temperature divides QK^T and sampling temperature divides the output logits, where is each one applied?

What is the goal and risk of injecting noise into pre-softmax attention scores?

Your team debates 32K vs 128K vs 256K vocab. What is the core tradeoff they should frame?

PremiumWhat's the 'attention sink'…

Why is softmax used in attention rather than alternatives like sparsemax or simple sum normalization?

Complete the canonical attention formula: Attention(Q, K, V) = ___(QKᵀ / ___) · V

Where in attention is FP32 still used, and what breaks if you push everything to FP16/BF16?