Transformers

Identify the attention-pattern difference between Mistral 7B and Llama-2 7B

Decode the acronym MHA in the transformer attention context.

Compare hidden_size, d_model, and embedding_dim, three names for the same thing or three different things?

Head-pruning studies report that many attention heads can be removed with negligible loss. Pick the correct interpretation.

Pick the formula that defines head_dim from hidden_size and num_heads.

Match GPT-2 versus Llama-2 attention design choices to their differences

How does torch.nn.functional.scaled_dot_product_attention relate to FlashAttention?

Why is causal attention cheaper than full bidirectional at the same length?

Name two production KV-cache eviction policies and what each prioritizes

Encoder self-attention versus decoder self-attention, what is the single structural difference at the mask level?

Inside a T5-style encoder-decoder, which sub-layer reads the encoder outputs, and through which projection?

Walk through how a token's d_model embedding is split into per-head pieces inside multi-head attention.

Where inside the attention sub-layer is dropout typically applied during training, and what does dropping there teach the model?

Walk through a T5 decoder block in order, list the attention sub-layers and what each one's Q, K, V read from.

Describe how continuous batching interacts with the KV cache

Identify the context vector inside scaled dot-product attention and what produces it.

Critique this claim about CLIP using cross-attention between towers

What problem does chunked prefill solve in production LLM serving?

How does prefix KV-cache sharing reduce serving cost across requests?

Name the additive bias term in attention and the two things it most commonly carries.

Injecting noise into pre-softmax attention scores, goal and risk?

Contrast ALiBi's linear distance bias with RoPE's vector rotation, which one modifies scores additively, and which one modifies Q and K directly?