Which pooling choice is standard when an embedding model keeps a decoder-only LLM's causal mask?

Encoder models like BERT use bidirectional attention: every token can attend to every other token, so every position has seen the full input context. For BERT-family embedders, mean pool (averaging all token vectors) works because all token vectors are computed from the same complete view of the sentence. **Decoder-only LLMs** (large language models) are different. They use a causal mask: token i can only attend to tokens 0 through i. Token 0 has seen only itself. Token 31 has seen all 32 positions. The vectors at different positions have very different context coverage. If you mean-pool over those vectors, you are averaging a token that has seen one position with a token that has seen the whole input. The low-context early positions blur the result. The usual choice is **last-token pooling**: take the hidden state at the final real (non-padding) position. That position has attended to the full sequence under causal masking, so its vector is a sensible sentence representation. Decoder-only embedders like e5-mistral-7b-instruct and Linq-Embed-Mistral use last-token pool. They often pair it with an instruction prefix (e.g. 'Instruct: retrieve relevant passages. Query: ...') so the last token is biased to compress the right thing.

**Causal masking** shapes the pooling choice. In a decoder-only LLM, the attention pattern is $$\text{mask}_{ij} = \begin{cases} 0 & i \geq j \\ -\infty & i < j \end{cases}$$ which means token i attends to positions 0 through i only. The visibility per position grows linearly from 1 (token 0 sees only itself) to N (token N-1 sees the full sequence). These vectors are not comparable. They encode different amounts of context. Mean pool collapses across these positions and gets a polluted average. Token 0's vector has roughly no context information. Yet it gets equal weight to token N-1's vector, which has full context. Mean pool still works in practice, but it does not remove this mismatch. **Last-token pool** takes the vector at the final real position, the only position that has attended to the full sequence. It is the natural analog of mean pool for the decoder-only architecture. e5-mistral-7b-instruct (Microsoft, 2024) and Linq-Embed-Mistral (Linq, 2024) use it. These models often pair last-token pool with an instruction prefix that biases the last token to compress the right semantic content for the downstream task. The other options fail. Averaging does not remove causal mask artifacts; it mixes vectors with mismatched context. CLS at position 0 has seen only itself, the worst possible position. First-token pool fails the same way. Decoder-only LLMs do not have a pretrained 'sequence-summary' position. Only the last position has attended to the full context.