**Encoder-only** transformers like BERT and the 2024 ModernBERT family stack only the encoder half of the original 2017 architecture. Every token attends to every other token in both directions, and the model outputs one hidden vector per input position. There is no decoder, no causal mask, and no way to generate the next token.
This shape is excellent at one specific job: turning a piece of text into a fixed-size representation that downstream code can use. Two flavors of that job dominate in 2026.
First, **dense retrieval embeddings**. A vector database stores one embedding per document chunk. At query time you embed the user query and run a nearest-neighbor search. Encoder-only models produce these embeddings faster and at higher quality per parameter than asking a decoder LLM for its hidden state. ModernBERT at ~150M parameters outperforms most retrieval setups that use a 7B decoder LLM as the embedder.
Second, **reranker base models**. A cross-encoder reranker reads the query and a candidate document together and outputs a relevance score. Encoder-only is the right base because you need full bidirectional attention over both inputs and you only need a single number out.
What encoder-only models cannot do: chat, write essays, reason step by step, call tools, or generate any text at all. Those are all decoder jobs.
**The shape determines the job.** Encoder-only transformers stack only the encoder half of the 2017 architecture: every layer is bidirectional self-attention, no causal mask, no generation head. The output is one hidden vector per input position, which you then pool (CLS token or mean pool) into a single fixed-size representation.
BERT (2018) introduced this design for **masked-language-model pretraining** plus a CLS token for classification. The bidirectional context is the load-bearing feature: token t can attend to every token before and after it, which produces richer per token representations than a decoder's left-to-right context can offer.
In 2026, encoder-only still dominates two roles:
1. **Dense retrieval embedders.** ModernBERT (~150M params, 8k context, 2024) is the modern reference. It produces 768-dim embeddings that beat embedding setups using 7B decoder LLMs as encoders, at a fraction of the latency and memory cost. Every production vector database (Pinecone, Weaviate, Qdrant) ships pipelines that default to an encoder-only embedder.
2. **Cross-encoder rerankers.** A reranker reads `[query, candidate_doc]` together and emits a relevance score. Bidirectional attention across the joined input is exactly what encoder-only provides. Decoder LLMs as rerankers are slower and rarely meaningfully better.
The distinguishing question for the multiple-choice options is generation. Chat, long-form writing, and chain-of-thought reasoning all require autoregressive generation. An encoder-only model has no generation head and no causal-attention path. It can only consume text and produce a fixed-size representation. So the only option that fits the encoder-only sweet spot is dense retrieval: consume text, produce a 768-dim vector, no generation needed.
Quick reference
TL;DR
Encoder-only transformers like BERT and ModernBERT still own dense retrieval and reranker bases in 2026.
Imagine two specialists. One is a librarian who reads any document from start to finish and then writes a single short fingerprint that summarizes the whole thing. The other is a novelist who can keep typing forever, one word at a time, picking up cues from everything written so far. For the question 'turn this paragraph into a number you can store in a database and search later', you want the librarian. They read both directions at once, they are fast, and they specialize in producing one tidy fingerprint per input. For the question 'write me a thousand-word reply that responds to my last three messages', you want the novelist. The librarian was never trained to generate; ask them to write a reply and they have nothing to give you. Encoder-only models are the librarian. Decoder-only LLMs are the novelist. Pick by the question, not by which one is fancier in 2026.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Any task that needs to generate text, code, or structured output Encoder-only has no generation head and no causal-attention path. Generation requires a decoder.
Long-form reasoning that benefits from chain-of-thought scratchpads Chain-of-thought is autoregressive token streaming. Decoder-only is the only viable shape.
Multi-turn chat with tool use and agent loops Each turn depends on the model's prior generation; only a decoder can produce that stream of turns.
Very small classification jobs where a 1B parameter model is overkill Even encoder-only is too heavy. A logistic regression on TF-IDF or a tiny distilled classifier is enough.
Quick scan · 2–4 min
Everything important, quickly.
4 min: encoder-only shape and bidirectional attention, MLM pretraining, why ModernBERT dominates retrieval, why decoder LLMs lose on cost, what encoder-only fundamentally cannot do.
Side-by-side
Property
Encoder-only (BERT, ModernBERT)
Decoder-only (GPT, Llama, Claude)
Attention
Bidirectional, no causal mask
Causal, left to right only
Generation
Cannot generate tokens
Autoregressive generation
Typical output shape
(batch, seq_len, d_model) hidden states pooled to one vector
Token stream
Pretraining objective
Masked-language modeling (MLM)
Next-token prediction (NTP)
Param range in 2026
~30M to ~500M (ModernBERT, bge-reranker)
1B to 1T+ (Llama 4, GPT-5.5)
Primary 2026 use
Retrieval embeddings, rerankers, classifiers
Chat, reasoning, agents, code, creative
Real-world examples
Real products, models, and research that use this idea.
1ModernBERT (Answer.AI, 2024) is the 2026 reference encoder-only embedder: ~150M params, 8k context, beats many 7B decoder embedders on retrieval benchmarks.
2bge-reranker-v3 (BAAI, 2024-2025) is an encoder-only cross-encoder reranker used widely in production RAG pipelines.
3Cohere Rerank 3 ships an encoder-only reranker as its production endpoint; the speed and cost story would be impossible with a decoder LLM base.
4Pinecone, Weaviate, Qdrant, and Vespa all default their first-pass embedding pipelines to encoder-only models even when their users are building decoder-LLM-powered chat apps on top.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does ModernBERT at 150M parameters beat a 7B decoder LLM as a retrieval embedder?
Sign in or go Pro to see the approach for this follow-up.
QCould you fine-tune a decoder LLM to do retrieval just as well as ModernBERT?
Sign in or go Pro to see the approach for this follow-up.
QWhy has no major lab released an 'encoder-only frontier model' in 2026 the way they release decoder-only frontier models?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating 'encoder-only' as just an older or weaker version of a decoder LLM. It is a different shape that produces bidirectional fixed-size representations and cannot generate at all.