Positional Encoding

What is the context window of an LLM?

How does a Vision Transformer turn an image into a sequence, and what mask runs over it?

Define T5's relative position bias and what bucketing buys you

Llama-3 bumps RoPE base from 10k to 500k, which long context property follows?

RoPE, spell out the acronym and name exactly which vectors it rotates

Contrast absolute and relative positional encoding by what the score depends on

Spot the flaw: a claim that attention by itself notices position.

Contrast ALiBi's linear distance bias with RoPE's vector rotation, which one modifies scores additively, and which one modifies Q and K directly?

ALiBi, decode the acronym and pinpoint where in the pipeline it injects position

Walk through Vaswani 2017's sinusoidal positional encoding, what's the formula and what property does it give?

Why has RoPE become the dominant positional encoding in modern LLMs?

How does RoPE differ structurally from sinusoidal positional encoding?

PremiumSpot the error in…

Why is positional encoding necessary in transformer architectures?

What's the practical tradeoff between learned and sinusoidal positional embeddings?

How does ALiBi inject position information without using positional embeddings?