Transformers

Where does cross-attention live in OpenAI Whisper?

What do you gain and lose by tying K and V to one projection?

What does dividing pre-softmax attention scores by an extra factor > 1 do at inference?

What is T5's relative position bias, and why is bucketing used?

Walk through KV-cache updates when speculative decoding rejects some draft tokens

Along which axis of the QK^T score matrix is softmax applied inside attention?

PremiumExplain effective context for…

How does softmax turn an attention score into an attention weight?

Name the scalar that QK^T is divided by inside scaled dot product attention.

What does RMSNorm keep from LayerNorm, and what does it drop?

Name what wraps the attention sub-layer in every transformer block.

Compare absolute and relative positional encoding. What does the score depend on in each case?

Fill the blank: for a single head, QK^T has shape (T_query, ___).

PremiumWhy does normalizing Q…

Which of pre-norm and post-norm makes deep stacks stable?

A batch packs a causal LM with left padded sequences and applies only the causal mask. Spot the mistake.

What does a padding mask zero out, and why is it needed when batching variable length sequences?

Spot the masking bug in this packed sequence training setup

Given Q of shape (B, n_heads, T, d_head), the per head attention output before concatenation has shape ___.

Describe the W_O projection in multi-head attention, its shape and what it mixes.

Beyond shape preservation, name two things W_O does after head concat

Which config token names the count of parallel attention heads in a layer?

Attention captures positional information automatically because the query at position 5 attends more to nearby keys than to distant ones, so positional encoding is just a small refinement.

PremiumTrue or false: swapping…

PremiumWhy does MQA underperform…