Positional Encoding

What is the context window of an LLM?

How does a Vision Transformer turn an image into a sequence under its attention mask?

What is T5's relative position bias, and why is bucketing used?

What happens to the longest context Llama-3 can use when the RoPE base goes from 10k to 500k?

What does RoPE stand for, and which vectors does it rotate?

Compare absolute and relative positional encoding. What does the score depend on in each case?

Attention captures positional information automatically because the query at position 5 attends more to nearby keys than to distant ones, so positional encoding is just a small refinement.

Compare ALiBi's linear distance bias with RoPE's vector rotation. Which one modifies scores additively, and which modifies Q and K directly?

What does ALiBi stand for, and where does it add position in the attention pipeline?

Walk through Vaswani 2017's sinusoidal positional encoding. What's the formula? What property does it give the model?

Why has RoPE become the dominant positional encoding in modern LLMs?

How does RoPE differ structurally from sinusoidal positional encoding?

PremiumSpot the error in…

Why is positional encoding necessary in transformer architectures?

What's the practical tradeoff between learned and sinusoidal positional embeddings?

How does ALiBi inject position information without using positional embeddings?