Pair each transformer family with its canonical pretraining objective

Match pairs·Easy·4.0 · 0·~2 min·Relevant atGoogle OpenAI

Attempt it

Encoder-only (BERT, ModernBERT)

Decoder-only (GPT, Llama, Claude)

Encoder-decoder (T5, FLAN-T5)

Encoder-decoder cross-modal (Whisper)

**Each transformer family has a canonical pretraining objective designed for its attention pattern.** **Encoder-only (BERT, ModernBERT) -> masked language modeling (MLM).** Random tokens in the input are replaced with [MASK]. The model uses bidirectional attention (every token can see every other token) to predict the masked tokens from context on both sides. This works because the encoder has no causal mask, so a token in position 5 can attend to tokens at positions 1 through 4 AND 6 through 100 to make its prediction. **Decoder-only (GPT, Llama, Claude) -> next-token prediction.** The model reads tokens left to right and predicts the next token at every position, using only the tokens that preceded it. The causal mask blocks attention to future positions, which means the model cannot cheat by looking ahead. This is what makes the model useful for generation: at inference time, it predicts one token at a time using the same mechanism it learned during pretraining. **Encoder-decoder (T5, FLAN-T5) -> span-corruption denoising.** Random SPANS (not single tokens) in the input are replaced with sentinel tokens like `<extra_id_0>`. The encoder sees the corrupted sequence with bidirectional attention; the decoder generates the original spans one sentinel at a time. This combines bidirectional understanding (in the encoder) with autoregressive generation (in the decoder). **Encoder-decoder cross-modal (Whisper) -> supervised audio to text seq2seq.** The encoder reads a mel-spectrogram of audio with bidirectional attention; the decoder generates the corresponding text transcription autoregressively. Unlike the others, this is supervised pretraining on labeled audio-text pairs, not self-supervised on raw text. The architecture is encoder-decoder because the input modality (audio) is structurally different from the output modality (text). **Why the pairings are not arbitrary.** Each objective requires a specific attention pattern. MLM needs bidirectional context to fill in the blank from both sides. Next-token prediction needs a causal mask to avoid cheating. Span corruption needs an encoder for bidirectional reading and a decoder for autoregressive generation. Cross-modal seq2seq needs separate encoder and decoder because the modalities differ.

**Pairings.** - Encoder-only (BERT, ModernBERT) -> **masked language modeling**. Random 15% of input tokens are masked, the model uses bidirectional attention to predict them. - Decoder-only (GPT, Llama, Claude) -> **causal next-token prediction**. Every position predicts the next token using only past context, enforced by a triangular causal mask. - Encoder-decoder (T5, FLAN-T5) -> **span-corruption denoising**. Random spans (typically 15% of tokens in 3-token spans) are replaced with sentinel tokens, the decoder generates the original spans autoregressively. - Encoder-decoder cross-modal (Whisper) -> **supervised audio to text seq2seq**. Encoder reads mel-spectrogram with bidirectional attention, decoder generates text transcription token by token. **Why each pairing is canonical.** **Encoder-only + MLM.** The encoder's lack of a causal mask is the key feature; every token can attend to every other token in both directions. This bidirectional context is what lets MLM work. A token in position 5 can use information from positions 1-4 (left context) and 6-100 (right context) to predict what should be at position 5. Try MLM with a causal mask and the model could only use left context, which wastes half the signal. Try next-token prediction with bidirectional attention and the model would trivially see the answer (the next token is in its receptive field), so the objective becomes useless. The mask and objective have to match. **Decoder-only + next-token.** The causal mask makes next-token prediction self-supervised at scale: every training token is a prediction target, every previous token is context. The mask prevents the trivial cheat of attending to the future. This is the most data-efficient text objective because it produces L training signals from an L-token sequence (versus ~0.15L for MLM with 15% masking).

Quick reference

TL;DR

Each architecture family has a canonical objective that fits its attention mask. Encoder-only uses MLM (bidirectional). Decoder-only uses next-token (causal). Text encoder-decoder uses span corruption.

Memory aid

Easy to grasp

Think of each transformer family like a different kind of student with a different study method. The encoder-only student (BERT) reads the whole sentence at once with some words hidden behind sticky notes, then guesses what is under each sticky note, this is masked language modeling. The decoder-only student (GPT, Llama) reads one word at a time and tries to predict the next word, never peeking ahead, this is next-token prediction. The text encoder-decoder student (T5) reads a sentence with whole phrases hidden, then has to write out the missing phrases one by one in a separate notebook, this is span corruption. The cross-modal student (Whisper) listens to an audio recording and writes down what was said in text form, this is supervised audio-to-text. Each method matches the architecture's attention pattern, which is why the pairings are not arbitrary.

Key concepts

Pretraining objectives

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 min: state each pairing, explain why the objective matches the attention mask in each case, mention that decoder-only's sample-efficiency advantage drives its dominance at scale.

Real-world examples

Real products, models, and research that use this idea.

BERT-base (2018): 12-layer encoder pretrained with MLM on Wikipedia + BookCorpus. The model that established MLM as the canonical encoder-only objective.
Llama 3.1 (2024): 8B / 70B / 405B decoder-only models pretrained with causal next-token on 15T+ tokens. The current open-weight standard for decoder-only + next-token.
T5 (2020): encoder-decoder pretrained with span corruption (15% mask rate, mean span length 3). Established the text-to-text paradigm that FLAN-T5 and UL2 built on.
Whisper (2022): encoder-decoder pretrained with supervised seq2seq on 680k hours of (audio, transcript) pairs. Multitask via prompt tokens fed to the decoder.
ELECTRA (2020): encoder-only with replaced-token detection (RTD) instead of MLM. More sample-efficient than MLM but never displaced it as the canonical encoder-only objective.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does next-token prediction give more training signal per token than MLM?

QCould you train a decoder-only model with MLM?

QWhy does Whisper use mel-spectrograms instead of raw waveforms?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Assuming any architecture can use any objective. The objective has to match the attention mask: causal masks force next-token prediction, bidirectional masks enable MLM, encoder-decoder masks enable span corruption.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.