Transformers

Where does cross-attention live in OpenAI Whisper?

Tying K and V to one projection, what is gained and what is risked?

Dividing pre-softmax attention scores by an extra factor > 1 at inference does what?

Define T5's relative position bias and what bucketing buys you

Walk through KV-cache updates when speculative decoding rejects some draft tokens

Along which axis of the QK^T score matrix is softmax applied inside attention?

PremiumExplain effective context for…

How does softmax turn an attention score into an attention weight?

Name the scalar that QK^T is divided by inside scaled dot product attention.

RMSNorm versus LayerNorm, what is kept and what is dropped?

Name what wraps the attention sub-layer in every transformer block.

Contrast absolute and relative positional encoding by what the score depends on

Fill the blank: for a single head, QK^T has shape (T_query, ___).

PremiumWhy does normalizing Q…

Pre-norm versus post-norm: which placement makes deep stacks stable?

A batch packs a causal LM with right padded sequences and applies only the causal mask. Spot the mistake.

What does a padding mask zero out, and why is it needed when batching variable length sequences?

Spot the masking bug in this packed sequence training setup

Given Q of shape (B, n_heads, T, d_head), the per head attention output before concatenation has shape ___.

Describe the W_O projection in multi-head attention, its shape and what it mixes.

Beyond shape preservation, name two things W_O does after head concat

Which config token names the count of parallel attention heads in a layer?

Spot the flaw: a claim that attention by itself notices position.

PremiumTrue or false: swapping…

PremiumWhy does MQA underperform…