AI Researcher

Define tied embeddings and explain why Llama-3 chose to untie them

Defend untying the output head: what does an extra ~500M parameters at the tail of the stack buy?

Which design choices from the original 'Attention Is All You Need' are no longer standard?

Same block, two regimes: training parallelism versus inference autoregression

Chinchilla's lesson: how did it reshape architecture and training choices after 2022?

Complete the RMSNorm formula

Why does a deep transformer stack stop training if you remove the residual connections?

Describe the residual stream as a read/write surface that sublayers act on

Pair each transformer family with its canonical pretraining objective

Walk through why pre-norm trains stably past 100 layers while post-norm needs warmup tricks

Spot the error: 'Pre-norm and post-norm are interchangeable, just a stylistic choice'

Match each positional encoding scheme to where it enters the architecture

Predict the trainable parameter count for a Llama-style 7B decoder block by block

Select the correct comparisons of an MoE transformer versus a dense transformer at equal active parameters

What does the router inside a Mixture-of-Experts block actually do, per token?

Pick the statements that accurately describe how an MoE sublayer replaces a dense FFN

Spot the error: 'The FFN mixes information across tokens'

Why does the transformer block use LayerNorm or RMSNorm instead of BatchNorm?

Pick the recipe a 2026 frontier open-weight LLM block actually ships

Define d_model and name three architecture knobs that scale with it

Order the tail-end operations that turn the last residual stream into logits

What does 'position-wise' actually mean for the transformer FFN sublayer?

Which sublayer of a dense transformer block carries more parameters, attention or FFN?

Defend the 4x FFN expansion ratio, then explain why SwiGLU breaks the simple rule

Pick the 2026 task where an encoder-only transformer still beats a decoder-only LLM