Attention Mechanism

143 questions

The mechanism that lets each token attend to every other token. Self-attention, multi-head, scaled dot-product, MQA, GQA, FlashAttention, KV cache.

Sort

Questions

RMSNorm versus LayerNorm, what is kept and what is dropped?
Multiple ChoiceMedium·Qual 4.0
Inside a transformer, what is the residual stream and how does attention interact with it?
FlashcardEasy·Qual 4.0
Name what wraps the attention sub-layer in every transformer block.
FlashcardEasy·Qual 4.0
Contrast absolute and relative positional encoding by what the score depends on
Multiple ChoiceMedium·Qual 4.0
Fill the blank: for a single head, QK^T has shape (T_query, ___).
Fill in BlankEasy·Qual 4.0
Why does normalizing Q and K before the dot product help training at scale?
Short AnswerMedium·Qual 4.0
During decode, why is only Q computed for the new token while full K and V come from cache?
Multiple ChoiceEasy·Qual 4.0
Prefill and decode: name the two phases of LLM inference and say which one is compute-bound
FlashcardEasy·Qual 4.0
Pre-norm versus post-norm: which placement makes deep stacks stable?
Multiple ChoiceMedium·Qual 4.0
vLLM defaults to a page size of 16. Pick what makes 1 and 1024 worse choices.
Multiple ChoiceMedium·Qual 4.0
A batch packs a causal LM with right-padded sequences and applies only the causal mask. Spot the mistake.
Spot the ErrorMedium·Qual 4.0
What does a padding mask zero out, and why is it needed when batching variable-length sequences?
FlashcardEasy·Qual 4.0
Spot the masking bug in this packed-sequence training setup
Spot the ErrorMedium·Qual 4.0
Given Q of shape (B, n_heads, T, d_head), the per-head attention output before concatenation has shape ___.
Fill in BlankEasy·Qual 4.0
Describe the W_O projection in multi-head attention, its shape and what it mixes.
FlashcardEasy·Qual 4.0
Beyond shape preservation, name two things W_O does after head concat
Short AnswerMedium·Qual 4.0
Spot the GQA configuration error: num_attention_heads=32, num_key_value_heads=7.
Spot the ErrorMedium·Qual 4.0
Which config token names the count of parallel attention heads in a layer?
FlashcardEasy·Qual 4.0
Spot the flaw: a claim that attention by itself notices position.
Spot the ErrorMedium·Qual 4.0
DeepSeek trains models to predict K tokens per step. Pick how that stays causal.
Multiple ChoiceMedium·Qual 4.0
True or false: swapping a single-head attention layer for a 12-head one (same d_model) raises the parameter count.
Multiple ChoiceMedium·Qual 4.0
Why does MQA underperform GQA when both share KV across heads?
Short AnswerMedium·Qual 4.0
MQA, spell out the acronym and name exactly what gets shared across heads
FlashcardEasy·Qual 4.0
Identify the attention-pattern difference between Mistral 7B and Llama-2 7B
Multiple ChoiceMedium·Qual 4.0
Decode the acronym MHA in the transformer attention context.
FlashcardEasy·Qual 4.0

Showing 26–50 of 143

← Prev Next →

Attention Foundations (6 cards)

Atomic flashcards for this topic. Free for everyone. Pro syncs them into your spaced-repetition queue.

Write the scaled dot-product attention formula.

Attention(Q, K, V) = softmax(QKᵀ / √d_k) · V

Why divide by √d_k?

To keep dot-product variance stable across dimensions, preventing softmax saturation and gradient vanishing.

What does multi-head attention provide?

Parallel sub-projections with independent learned parameters, letting different heads specialize in different relationships (positional, syntactic, long range).

What is causal masking?

Setting future-position attention scores to -∞ before softmax so a token can only attend to itself and prior tokens, required for autoregressive language modeling.

+ 2 more cards in the full deck · Drill all cards (Pro)

Master this topic

Daily practice mixes topics. Topic specific drill is a Pro feature.

Start practice