Transformers

Where does cross-attention live in OpenAI Whisper?

Tying K and V to one projection, what is gained and what is risked?

Dividing pre-softmax attention scores by an extra factor > 1 at inference does what?

Define T5's relative position bias and what bucketing buys you

Walk through KV-cache updates when speculative decoding rejects some draft tokens

Along which axis of the QK^T score matrix is softmax applied inside attention?

Explain effective context for Mistral's 4096 sliding window over a 16k input

How does softmax turn an attention score into an attention weight?

Name the scalar that QK^T is divided by inside scaled dot-product attention.

RMSNorm versus LayerNorm, what is kept and what is dropped?

Name what wraps the attention sub-layer in every transformer block.

Contrast absolute and relative positional encoding by what the score depends on

Fill the blank: for a single head, QK^T has shape (T_query, ___).

Why does normalizing Q and K before the dot product help training at scale?

Pre-norm versus post-norm: which placement makes deep stacks stable?

A batch packs a causal LM with right-padded sequences and applies only the causal mask. Spot the mistake.

What does a padding mask zero out, and why is it needed when batching variable-length sequences?

Spot the masking bug in this packed-sequence training setup

Given Q of shape (B, n_heads, T, d_head), the per-head attention output before concatenation has shape ___.

Describe the W_O projection in multi-head attention, its shape and what it mixes.

Beyond shape preservation, name two things W_O does after head concat

Which config token names the count of parallel attention heads in a layer?

Spot the flaw: a claim that attention by itself notices position.

True or false: swapping a single-head attention layer for a 12-head one (same d_model) raises the parameter count.

Why does MQA underperform GQA when both share KV across heads?