Fill in Blank

Fill in the blanks: the special tokens that control sequence boundaries and conversation structure in modern LLMs

Complete the L2 normalization formula

Complete the dot product as cosine identity on unit vectors

When does cosine similarity equal the dot product?

Identify the most common pooling strategy in SBERT family encoder embeddings

Identify the key difference between Word2Vec and SBERT style embeddings

Complete the Reciprocal Rank Fusion (RRF) score formula

Complete the definition of Matryoshka representation learning

Complete the pre-norm transformer block.

Fill in the tensor shapes as a batch flows through one transformer block

Complete the per token KV cache memory formula

Complete the RMSNorm formula

Complete the path from token id to the vector that enters block 1

Complete the canonical attention formula: Attention(Q, K, V) = ___(QKᵀ / ___) · V

Fill in the scaled dot product attention formula.

Fill the blank: for a single head, QK^T has shape (T_query, ___).

Each row of the post-softmax attention weight matrix corresponds to which side of the QK product?

Name the two properties softmax guarantees for every row of the attention weight matrix

If d_model = 768 and num_heads = 12, what is d_head?

What are the typical head counts and head dimensions for modern 7B models?

Given Q of shape (B, n_heads, T, d_head), the per head attention output before concatenation has shape ___.

Complete the Kaplan scaling law trend statement

Fill the conditions that preserve accumulation to large batch equivalence

Fill in the full FT memory contributions for a 7B model

Fill in the memory delta when a 7B weight tensor moves from fp32 to bf16