Match each transformer family to the kind of attention it uses

Match pairs·Easy·4.0 · 0·~2 min·Asked atFlowise Hcl Runway·Relevant atAi4bharat Cerebras Microsoft Replicate

Attempt it

Drag each answer to line up with its matching prompt

Encoder-only (BERT, RoBERTa)

Bidirectional self-attention only; no causal mask, no cross-attention.

Decoder-only (GPT, Llama, Mistral)

Bidirectional self-attention within a designated prefix block, causal self-attention afterward.

Encoder-decoder (T5, BART)

Bidirectional self-attention in the encoder plus causal self-attention and cross-attention in the decoder.

Prefix LM (UL2, some T5 modes)

Causal self-attention only; no cross-attention, no bidirectional path.

Transformer families differ in three things: whether self-attention is causally masked, whether it is bidirectional, and whether cross-attention exists. Combining these three gives the four families: **Encoder-only (BERT, RoBERTa):** - Bidirectional self-attention (any token can attend to any other). - No causal mask. - No cross-attention. - Used for classification, embedding, masked-token prediction. **Decoder-only (GPT, Llama, Mistral):** - Causal self-attention (each token only sees previous tokens). - No cross-attention. - Used for autoregressive generation (chat, completion). **Encoder-decoder (T5, BART):** - Encoder: bidirectional self-attention. - Decoder: causal self-attention plus cross-attention into the encoder's output. - Used for translation, summarization, ASR. **Prefix LM (UL2, some T5 modes):** - Bidirectional self-attention over a designated prefix block. - Causal self-attention for everything after the prefix. - A hybrid that captures benefits of both. The family choice is task-driven. Classification benefits from bidirectional context (encoder-only). Generation needs causal masking (decoder-only). Source to target with reuse benefits from cross-attention (encoder-decoder).

Four transformer families, three structural switches. The switches are: causal masking on self-attention, bidirectional self-attention, and presence of cross-attention. **Encoder-only (BERT, RoBERTa, DeBERTa)** flips on bidirectional self-attention, no causal mask, no cross-attention. Every token attends to every other token. The objective is masked-token prediction or classification on top of contextualized embeddings. **Decoder-only (GPT-5.5, Llama 4 Maverick, Mistral, DeepSeek V4, Qwen 3.5)** flips on causal self-attention, no cross-attention. Each token attends only to previous tokens. The objective is next-token prediction. This is the dominant LLM family in 2026. **Encoder-decoder (T5, BART, Whisper)** combines both: a bidirectional encoder feeds a decoder that has causal self-attention plus cross-attention sub-layers reading from the encoder output. Used for translation, summarization, speech to text, workloads with a distinct source and target. **Prefix-LM (UL2, some T5 modes, prefix-tuning variants)** uses bidirectional attention within a designated prefix block, then switches to causal for the rest. The prefix's K, V can be cached like an encoder output. A compromise design that earns some bidirectional benefit without paying the full encoder-decoder complexity. The family choice is workload-driven, and decoder-only has won the general-purpose LLM market because its simplicity scales best across heterogeneous chat-style traffic.

TL;DR

Four families flip three switches: causal masking, bidirectional self-attention, and cross-attention. Each family is defined by which combination of these its blocks use.

Memory aid

Easy to grasp

Imagine four types of newsrooms. The first lets every reporter read every other reporter's notes freely, all at once, that's encoder-only (BERT). The second lets each reporter only read notes written before their own, that's decoder-only (GPT). The third has a research team that reads everything freely and a writing team that reads its own past plus the research team's report, that's encoder-decoder (T5). The fourth treats the first half of the day like the first newsroom and the second half like the second, that's prefix-LM. The flipping of two switches (can I look both ways, can I see the other team) gives you four newsrooms.

Key concepts

Encoder architecture Decoder architecture Causal mask Cross-attention Prefix LM

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

The transformer architecture family tree is one of those things that looks intimidating because of the proper-noun explosion (BERT, GPT, T5, BART, UL2, PaLM, Llama, Mistral, DeepSeek...) but compresses to four boxes once you internalize the three structural switches that distinguish them. Knowing this taxonomy is the difference between asking 'why does BERT not generate?' (because it has no causal mask) and 'why does Llama not have cross-attention?' (because it has no separate encoder).

This deep dive walks each family's structural choices, traces the historical evolution that produced each, and ends with the workload-driven decision guide for 2026.

The three switches

Every transformer family flips a combination of three switches.

Switch 1: causal mask on self-attention

OFF (bidirectional): every token attends to every other token. Used in models that compute contextualized embeddings without generation.

ON (causal): each token only attends to previous tokens. Required for autoregressive generation, the model cannot peek at tokens it has not yet generated.

Switch 2: cross-attention sub-layer

ABSENT: the architecture has only self-attention. All conditioning happens within one sequence.

PRESENT: a decoder reads from a separate encoder via cross-attention. The decoder's Q comes from itself; K and V come from the encoder. Information flows one-directionally from encoder to decoder.

Switch 3: hybrid mask within one stream (prefix-LM variant)

This is a sub-case of switch 1. Instead of one uniform mask across the whole sequence, the model uses bidirectional attention over a designated prefix block and causal attention after. The prefix is structurally like a mini-encoder embedded in the same stream.

The four canonical settings

Family	Switch 1	Switch 2	Switch 3
Encoder-only	OFF (bidirectional)	ABSENT	n/a
Decoder-only	ON (causal)	ABSENT	n/a
Encoder-decoder	encoder OFF, decoder ON	PRESENT in decoder	n/a
Prefix-LM	mixed	ABSENT	YES

Four families, three switches. Memorize the table and the entire ecosystem clicks.

Encoder-only: bidirectional context for understanding

Decoder-only: causal generation at scale

Encoder-decoder: source-target asymmetry

Prefix-LM: the compromise design

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Family	Self-attn mask	Cross-attn?	Primary task
Encoder-only	Bidirectional	No	Classification, retrieval
Decoder-only	Causal	No	Autoregressive generation
Encoder-decoder	Bidirectional (enc) + Causal (dec)	Yes	Translation, ASR, summarization
Prefix-LM	Bidirectional in prefix, causal after	No	Mixed generation + context

Real products, models, and research that use this idea.

BERT (encoder-only, 2018): bidirectional masked language modeling; still the basis for re-rankers and dense retrievers in 2026 production.
GPT-5.5, Claude Opus 4.7, Llama 4 Maverick, Gemini 3.1 Pro, DeepSeek V4, Qwen 3.5 (decoder-only, 2026): the dominant LLM family.

What an interviewer would ask next. Try answering before peeking at the approach.

QIf decoder-only models have won the chat market, why is encoder-only still used for retrieval and re-ranking?

Bidirectional context gives richer per-token embeddings for similarity tasks. Encoder-only models also do not need to generate, so they avoid the KV cache and autoregressive overhead entirely. For a re-ranking pass over 100 candidates, an encoder-only model is 5-10x cheaper than running a decoder-only model in scoring mode.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating encoder-only and the encoder of an encoder-decoder as different things. They use the same bidirectional self-attention; the difference is whether a decoder consumes their output.

60 second bullets to scan on the way to the call.

Three switches: causal mask, bidirectional self-attention, cross-attention presence
Encoder-only: bidirectional only, no cross-attention

Primary sources. Browse if you want the original framing.