Pick the only sound pooling choice when an embedding model is built from a decoder-only LLM

MCQ·Hard·4.0 · 0·~1 min·Asked atCanva Induced Ai Replicate

Attempt it

Encoder models like BERT use bidirectional attention: every token can attend to every other token, so every position has seen the full input context. For BERT-family embedders, mean pool works because all token vectors are computed from the same complete view of the sentence. **Decoder-only LLMs** are different. They use a causal mask: token i can only attend to tokens 0 through i. Token 0 has seen only itself. Token 32 has seen all 32 positions. The vectors at different positions have radically different context coverage. If you mean-pool over those vectors, you are averaging a token that has seen one position with a token that has seen the whole input. The result is incoherent, polluted by the no-context early positions. The fix is **last-token pooling**: take the hidden state at the final real (non-padding) position. That position has attended to the full sequence under causal masking, so its vector is a sensible sentence representation. Decoder-only embedders like e5-mistral-7b-instruct, NV-Embed-v2, and Linq-Embed-Mistral all use last-token pool. They often pair it with an instruction prefix (e.g. 'Instruct: retrieve relevant passages. Query: ...') so the last token is biased to compress the right thing.

**Causal masking** forces the pooling choice. In a decoder-only LLM, the attention pattern is $$\text{mask}_{ij} = \begin{cases} 0 & i \geq j \\ -\infty & i < j \end{cases}$$ which means token i attends to positions 0 through i only. The visibility per position grows linearly from 1 (token 0 sees only itself) to N (token N-1 sees the full sequence). These vectors are not comparable; they encode different amounts of context. Mean pool collapses across these positions and gets a polluted average. Token 0's vector has roughly no context information; it gets equal weight to token N-1's vector which has full context. The result is a sentence representation skewed toward the no-context positions, which damages retrieval quality. **Last-token pool** takes the vector at the final real position, the only position that has attended to the full sequence. It is the natural analog of mean pool for the decoder-only architecture. Used in 2026 by e5-mistral-7b-instruct (Microsoft, 2024), NV-Embed-v2 (NVIDIA, 2024), Linq-Embed-Mistral (Linq, 2024), GritLM-7B. These models often pair last-token pool with an instruction prefix that biases the last token to compress the right semantic content for the downstream task. The other options fail. Mean pool over causal hidden states mixes context-mismatched vectors. CLS at position 0 has seen only itself, the worst possible position. First-token pool is the same trap. Decoder-only LLMs do not have a pretrained 'sequence-summary' position; the only position that has the structural property of full-context attendance is the last one.

TL;DR

Decoder-only embedders use last-token pool because causal masking gives only the final position full-sequence visibility; mean-pooling mixes vectors with mismatched context coverage.

Memory aid

Easy to grasp

Picture an assembly line where each worker only sees the parts that came before them. Worker 1 has seen one part. Worker 32 has seen everything. If you ask the line for a summary, you ask worker 32, the one who has seen the whole product. Averaging all the workers' opinions is silly because worker 1 was answering about one part while worker 32 was answering about the whole thing. Last-token pool asks worker 32. Mean pool averages every worker, which mixes apples and oranges.

Key concepts

Embeddings

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Decoder-only LLMs have become the dominant pretraining substrate (GPT-5.5, Claude Opus 4.7, Llama 4 Maverick, DeepSeek V4, Qwen 3.5), and naturally a question arose: can we use these models as embedding backbones? The answer is yes, but it forces a different pooling choice than the encoder-family default. The architectural argument is the question.

This deep dive covers the causal mask and what it does to per-position context coverage, why mean pool fails in this setting, why last-token pool is the natural primitive, the instruction-prefix interaction that makes contrastive fine-tuning work, the 2026 decoder-only embedder landscape, and the LLM2Vec escape hatch that lets you use mean pool by removing the causal mask.

Mental model: in a decoder, only the last token has seen the whole sequence. Pooling that last position is the only sound aggregation; everything else mixes vectors with mismatched context coverage.

The causal mask and per-position context coverage

What the causal mask does

A decoder-only transformer applies a causal (lower-triangular) attention mask:

\text{mask}_{ij} = \begin{cases} 0 & i \geq j \\ -\infty & i < j \end{cases}

This means token i can attend to positions 0 through i. Future positions (j > i) are masked out. The mask is what makes autoregressive generation work, the model cannot peek at future tokens.

Per-position context coverage

Define context coverage c(i) as the number of positions token i has attended to. Under causal masking:

Token 0: c(0) = 1 (only itself).
Token 1: c(1) = 2 (itself plus one prior).
...
Token N-1: c(N-1) = N (the full sequence).

The coverage grows linearly across positions. This is the load-bearing fact for the pooling argument.

Contrast with encoder bidirectional attention

In an encoder (BERT, RoBERTa), every position attends to every other position. c(i) = N for every i. All token vectors are computed from the full input. Mean pool works because the vectors are interchangeable views of the same content.

In a decoder, the vectors are NOT interchangeable. Token 0's vector has effectively no context information; token N-1's vector has full context. Averaging them is a category error.

Why mean pool fails on decoder hidden states

Last-token pool and the instruction-prefix interaction

The LLM2Vec escape hatch and the senior takeaway

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

e5-mistral-7b-instruct (Microsoft, 2024) introduced the decoder-only last token with instruction pattern at scale.
NV-Embed-v2 (NVIDIA, 2024) tops MTEB and uses last-token pooling on a Mistral-7B base.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat does removing the causal mask and re-finetuning a decoder buy you?

It converts the decoder into a bidirectional encoder, enabling mean pool. Models like LLM2Vec do this. Cost: you break generative capability and need extra fine-tuning compute. Benefit: better embedding quality in some cases, but the gain is small and the engineering is invasive.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Reaching for mean pool because it is the encoder default. Mean pool requires bidirectional context, which decoder-only models do not provide; the early-position vectors are not comparable to the late-position ones.

60 second bullets to scan on the way to the call.

What the causal mask does to attention in a decoder-only LLM
Per-position context coverage in a decoder vs an encoder

Primary sources. Browse if you want the original framing.