AI Researcher

Encoder self-attention versus decoder self-attention, what is the single structural difference at the mask level?

Match each transformer family to the kind of attention it uses

Inside a T5-style encoder-decoder, which sub-layer reads the encoder outputs, and through which projection?

Translating one source sentence into many target languages, what attention-side trick scales?

Walk through how a token's d_model embedding is split into per-head pieces inside multi-head attention.

Where inside the attention sub-layer is dropout typically applied during training, and what does dropping there teach the model?

What does the dot product q . k actually measure inside an attention layer?

Decoder-only LLMs ship without a cross-attention sub-layer, so where does the context go?

Walk through a T5 decoder block in order, list the attention sub-layers and what each one's Q, K, V read from.

Identify the context vector inside scaled dot-product attention and what produces it.

BERT prepends a [CLS] token to every input, what is its actual job in the model?

Critique this claim about CLIP using cross-attention between towers

For a causal mask over a 4-token sequence, what value sits at row 2, column 3?

Find the bug: a Llama deployment serves traffic without prepending the BOS token.

Name the additive bias term in attention and the two things it most commonly carries.

In a 7B transformer, do attention or MLP layers eat more FLOPs, and does context length change the answer?

After W_O projects the multi-head output back to d_model, what happens before the next sub-layer runs?

Name two recovery moves a serving stack can pull when KV memory runs short mid-request.

Injecting noise into pre-softmax attention scores, goal and risk?

Define a single attention head, what does one head actually own and produce?

Contrast ALiBi's linear distance bias with RoPE's vector rotation, which one modifies scores additively, and which one modifies Q and K directly?

ALiBi, decode the acronym and pinpoint where in the pipeline it injects position

A model card reports perplexity of 3.2. Explain what that number means in plain English.

You suspect your custom eval set leaked into a model's training data. Describe two practical tests you would run to detect contamination.

TruthfulQA is designed to catch a specific model failure. Describe what it tests and why larger models sometimes do worse.