Choose the positional scheme that fails hardest when context quadruples at inference

MCQ·Medium·4.0 · 0·~1 min·Asked atInduced Ai Perplexity Snap

Attempt it

**Positional schemes split into table-based and function-based.** Learned absolute embeddings are a parameter table with one row per possible position; everything else is a formula that takes a position as input and computes a vector or bias. When you try to run a 4k-trained model on 16k tokens at inference, the table approach hits a wall. There is no row 4097. The model literally cannot compute what to feed in. The other schemes keep producing output: - **Sinusoidal encodings** use `sin` and `cos` of `pos / 10000^(2i/d)`. Defined for any position. - **RoPE** rotates Q and K by `pos`-dependent angles. The rotation is always computable; quality may degrade because the model was not trained on such large angles, but it does not crash. - **ALiBi** subtracts a head-specific slope times `|i - j|` from attention logits. The bias is well-defined at any distance and was specifically designed to extrapolate. This structural difference is why frontier LLMs abandoned learned absolute positions. Llama, Mistral, DeepSeek, Qwen, and Gemma all use RoPE; BLOOM and MPT use ALiBi. Nobody is shipping a learned-absolute frontier decoder in 2026.

**The structural failure of learned absolute embeddings.** A learned absolute scheme adds a trainable `(max_seq_len, d_model)` table to the model. The forward pass does `x = token_embed(token_id) + pos_embed(position)`. Position 4097 of a 4k-trained model has no row in that table, so the lookup is undefined. To run past training length, you have to either truncate to 4k, randomly initialize new rows (terrible quality), or copy-and-perturb existing rows (only slightly less terrible). **Why the others extrapolate.** Sinusoidal PE is a closed-form function of position; `sin(pos / 10000^(2i/d))` is defined for any `pos`. ALiBi adds a per-head linear bias to attention logits proportional to `|i - j|`; again, well-defined at any distance. RoPE rotates Q and K by position-dependent angles before the dot product; the rotation matrix is computable at any position. None of these extrapolate perfectly. RoPE quality drops once the rotation angles at long positions exceed what was seen during training; this is what theta rescaling, NTK scaling, and YaRN address. But the model produces output, and the quality degradation is gradual. Learned absolute has no graceful-degradation path because there is nothing to degrade gracefully from. **Why this matters in practice.** The 2026 long-context recipe is RoPE plus theta rescaling plus continued pretraining on long sequences (Llama 3.1 extended from 8k to 128k this way). The reason RoPE won is exactly this extrapolation property; learned absolute simply cannot participate in that workflow.

TL;DR

Learned absolute embeddings are a lookup table with exactly `max_seq_len` rows. Position 4097 of a 4k model has no row at all.

Memory aid

Easy to grasp

Imagine four ways of telling a chef where on the plate to put a garnish. Three of them are formulas: measure from the edge, count clockwise, or use a compass. Those work for any plate size. The fourth method is a stamp book with exactly one stamp per position on a 12-inch plate. Hand the chef a 48-inch plate and the stamp book is useless past the 12-inch mark. There is no entry. The other three formulas keep producing answers; they may not be perfectly tuned, but they extrapolate. That is exactly why frontier LLMs dropped the stamp book in favor of formulas.

Key concepts

Transformer architecture

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

The question looks like a knowledge check on four positional schemes, but it is really a structural test: do you see that learned absolute is in a different category from the other three? One is a parameter table; three are functions of position. The implications for inference-time context extension are completely different.

This distinction explains why frontier LLMs converged on RoPE plus theta rescaling rather than any learned-absolute variant, and why long-context extension workflows simply do not exist for the learned-absolute branch.

Tables vs functions: the structural divide

A positional scheme provides each token with some signal of its position. The schemes split into two structural categories.

Parameter tables. Learned absolute embeddings store a (max_seq_len, d_model) matrix of parameters. Position p is looked up as row p of this table. The lookup is undefined for p >= max_seq_len. The table size is a hard cap baked into the model's parameter count.

Functions of position. Sinusoidal PE computes sin(pos / 10000^(2i/d)) and cos(...) from the position and the dimension index. RoPE computes rotation angles theta_m = m / 10000^(2i/d) and applies them to Q and K. ALiBi computes a per-head bias -m_h * |i - j| and adds it to attention logits. All three take pos (or pairs of positions) as input and produce output for any value.

The table approach has a hard failure at max_seq_len. The function approach degrades gradually past training length because the model has not seen those input values, but it still produces something. This is the only structural difference that matters for the question.

The mechanics of the learned-absolute failure

How the function-based schemes extend

Why this drove frontier convention

Putting numbers to it: 2026 frontier-model context

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Llama 3 used RoPE with theta = 500000, up from Llama 2's 10000, to support 128k context after long-sequence continued pretraining.
Llama 3.1 8B and 70B extended from an 8k base to 128k via theta rescaling plus long-context fine-tuning, exactly the workflow learned absolute cannot support.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy did Llama 3 raise theta from 10000 to 500000 instead of using YaRN or PI?

Plain theta rescaling with sufficient long-context continued pretraining can match more elaborate recipes like YaRN at the cost of compute, and it is simpler to implement. The choice is partly a budget call.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Saying ALiBi fails because its bias is undefined past training length. It is defined for every distance and was designed precisely for extrapolation.

60 second bullets to scan on the way to the call.

Define learned absolute, sinusoidal, RoPE, and ALiBi in one sentence each
Explain why a learned absolute table cannot produce a vector for position max_seq_len + 1

Primary sources. Browse if you want the original framing.