Match each transformer family to the kind of attention it uses

Match pairs·Easy·4.0 · 0·~2 min

Attempt it

Encoder-only (BERT, RoBERTa)

Decoder-only (GPT, Llama, Mistral)

Encoder-decoder (T5, BART)

Prefix LM (UL2, some T5 modes)

Transformer families differ in three things: whether self-attention is causally masked, whether it is bidirectional, and whether cross-attention exists. Combining these three gives the four families: **Encoder-only (BERT, RoBERTa):** - Bidirectional self-attention (any token can attend to any other). - No causal mask. - No cross-attention. - Used for classification, embedding, masked-token prediction. **Decoder-only (GPT, Llama, Mistral):** - Causal self-attention (each token only sees previous tokens). - No cross-attention. - Used for autoregressive generation (chat, completion). **Encoder-decoder (T5, BART):** - Encoder: bidirectional self-attention. - Decoder: causal self-attention plus cross-attention into the encoder's output. - Used for translation, summarization, ASR. **Prefix LM (UL2, some T5 modes):** - Bidirectional self-attention over a designated prefix block. - Causal self-attention for everything after the prefix. - A hybrid that captures benefits of both. The family choice is task-driven. Classification benefits from bidirectional context (encoder-only). Generation needs causal masking (decoder-only). Source to target with reuse benefits from cross-attention (encoder-decoder).

Four transformer families, three structural switches. The switches are: causal masking on self-attention, bidirectional self-attention, and presence of cross-attention. **Encoder-only (BERT, RoBERTa, DeBERTa)** flips on bidirectional self-attention, no causal mask, no cross-attention. Every token attends to every other token. The objective is masked-token prediction or classification on top of contextualized embeddings. **Decoder-only (GPT-5.5, Llama 4 Maverick, Mistral, DeepSeek V4, Qwen 3.5)** flips on causal self-attention, no cross-attention. Each token attends only to previous tokens. The objective is next-token prediction. This is the dominant LLM family in 2026. **Encoder-decoder (T5, BART, Whisper)** combines both: a bidirectional encoder feeds a decoder that has causal self-attention plus cross-attention sub-layers reading from the encoder output. Used for translation, summarization, speech to text, workloads with a distinct source and target. **Prefix-LM (UL2, some T5 modes, prefix-tuning variants)** uses bidirectional attention within a designated prefix block, then switches to causal for the rest. The prefix's K, V can be cached like an encoder output. A compromise design that earns some bidirectional benefit without paying the full encoder-decoder complexity. The family choice is workload-driven, and decoder-only has won the general-purpose LLM market because its simplicity scales best across heterogeneous chat-style traffic.

Quick reference

TL;DR

Four families flip three switches: causal masking, bidirectional self-attention, and cross-attention. Each family is defined by which combination of these its blocks use.

Memory aid

Easy to grasp

Imagine four types of newsrooms. The first lets every reporter read every other reporter's notes freely, all at once, that's encoder-only (BERT). The second lets each reporter only read notes written before their own, that's decoder-only (GPT). The third has a research team that reads everything freely and a writing team that reads its own past plus the research team's report, that's encoder-decoder (T5). The fourth treats the first half of the day like the first newsroom and the second half like the second, that's prefix-LM. The flipping of two switches (can I look both ways, can I see the other team) gives you four newsrooms.

Key concepts

Encoder architecture Decoder architecture Causal mask Cross-attention Prefix LM

Detailed answer & concept explanation

~7 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

Name the three switches, walk each family's switch settings, give one example model per family, and close with why decoder-only dominates general-purpose LLMs in 2026 while encoder-only and encoder-decoder retain specific niches.

Side-by-side

Family	Self-attn mask	Cross-attn?	Primary task
Encoder-only	Bidirectional	No	Classification, retrieval
Decoder-only	Causal	No	Autoregressive generation
Encoder-decoder	Bidirectional (enc) + Causal (dec)	Yes	Translation, ASR, summarization
Prefix-LM	Bidirectional in prefix, causal after	No	Mixed generation + context

Real-world examples

Real products, models, and research that use this idea.

BERT (encoder-only, 2018): bidirectional masked language modeling; still the basis for re-rankers and dense retrievers in 2026 production.
GPT-5.5, Claude Opus 4.7, Llama 4 Maverick, Gemini 3.1 Pro, DeepSeek V4, Qwen 3.5 (decoder-only, 2026): the dominant LLM family.
T5 (encoder-decoder, 2019): the canonical text to text design with cross-attention bridging encoder and decoder.
Whisper (encoder-decoder ASR, 2022): audio encoder + text decoder with cross-attention, still the production ASR baseline.
UL2 (prefix-LM, 2022): mixed bidirectional then causal training; demonstrated the prefix-LM design at scale.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QIf decoder-only models have won the chat market, why is encoder-only still used for retrieval and re-ranking?

QCould a prefix-LM be used to bring back some of the encoder-decoder's source-reuse benefits to a decoder-only style architecture?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating encoder-only and the encoder of an encoder-decoder as different things. They use the same bidirectional self-attention; the difference is whether a decoder consumes their output.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.