AI Researcher

RoPE, spell out the acronym and name exactly which vectors it rotates

RMSNorm versus LayerNorm, what is kept and what is dropped?

Inside a transformer, what is the residual stream and how does attention interact with it?

Name what wraps the attention sub-layer in every transformer block.

Contrast absolute and relative positional encoding by what the score depends on

Fill the blank: for a single head, QK^T has shape (T_query, ___).

Why does normalizing Q and K before the dot product help training at scale?

During decode, why is only Q computed for the new token while full K and V come from cache?

Pre-norm versus post-norm: which placement makes deep stacks stable?

A batch packs a causal LM with right-padded sequences and applies only the causal mask. Spot the mistake.

What does a padding mask zero out, and why is it needed when batching variable-length sequences?

Given Q of shape (B, n_heads, T, d_head), the per-head attention output before concatenation has shape ___.

Describe the W_O projection in multi-head attention, its shape and what it mixes.

Beyond shape preservation, name two things W_O does after head concat

Which config token names the count of parallel attention heads in a layer?

Spot the flaw: a claim that attention by itself notices position.

DeepSeek trains models to predict K tokens per step. Pick how that stays causal.

True or false: swapping a single-head attention layer for a 12-head one (same d_model) raises the parameter count.

Why does MQA underperform GQA when both share KV across heads?

Decode the acronym MHA in the transformer attention context.

Compare hidden_size, d_model, and embedding_dim, three names for the same thing or three different things?

Head-pruning studies report that many attention heads can be removed with negligible loss. Pick the correct interpretation.

Pick the formula that defines head_dim from hidden_size and num_heads.

Modern 7B-class LLMs, fill in typical head counts and the resulting per-head dimension

Does FlashAttention change the attention output relative to standard attention?