Pick the only sound pooling choice when an embedding model is built from a decoder-only LLM

Encoder models like BERT use bidirectional attention: every token can attend to every other token, so every position has seen the full input context. For BERT-family embedders, mean pool works because all token vectors are computed from the same complete view of the sentence. **Decoder-only LLMs** are different. They use a causal mask: token i can only attend to tokens 0 through i. Token 0 has seen only itself. Token 32 has seen all 32 positions. The vectors at different positions have radically different context coverage. If you mean-pool over those vectors, you are averaging a token that has seen one position with a token that has seen the whole input. The result is incoherent, polluted by the no-context early positions. The fix is **last-token pooling**: take the hidden state at the final real (non-padding) position. That position has attended to the full sequence under causal masking, so its vector is a sensible sentence representation. Decoder-only embedders like e5-mistral-7b-instruct, NV-Embed-v2, and Linq-Embed-Mistral all use last-token pool. They often pair it with an instruction prefix (e.g. 'Instruct: retrieve relevant passages. Query: ...') so the last token is biased to compress the right thing.

**Causal masking** forces the pooling choice. In a decoder-only LLM, the attention pattern is $$\text{mask}_{ij} = \begin{cases} 0 & i \geq j \\ -\infty & i < j \end{cases}$$ which means token i attends to positions 0 through i only. The visibility per position grows linearly from 1 (token 0 sees only itself) to N (token N-1 sees the full sequence). These vectors are not comparable; they encode different amounts of context. Mean pool collapses across these positions and gets a polluted average. Token 0's vector has roughly no context information; it gets equal weight to token N-1's vector which has full context. The result is a sentence representation skewed toward the no-context positions, which damages retrieval quality. **Last-token pool** takes the vector at the final real position, the only position that has attended to the full sequence. It is the natural analog of mean pool for the decoder-only architecture. Used in 2026 by e5-mistral-7b-instruct (Microsoft, 2024), NV-Embed-v2 (NVIDIA, 2024), Linq-Embed-Mistral (Linq, 2024), GritLM-7B. These models often pair last-token pool with an instruction prefix that biases the last token to compress the right semantic content for the downstream task. The other options fail. Mean pool over causal hidden states mixes context-mismatched vectors. CLS at position 0 has seen only itself, the worst possible position. First-token pool is the same trap. Decoder-only LLMs do not have a pretrained 'sequence-summary' position; the only position that has the structural property of full-context attendance is the last one.