Pair each transformer family with its canonical pretraining objective

**Each transformer family has a canonical pretraining objective designed for its attention pattern.** **Encoder-only (BERT, ModernBERT) -> masked language modeling (MLM).** Random tokens in the input are replaced with [MASK]. The model uses bidirectional attention (every token can see every other token) to predict the masked tokens from context on both sides. This works because the encoder has no causal mask, so a token in position 5 can attend to tokens at positions 1 through 4 AND 6 through 100 to make its prediction. **Decoder-only (GPT, Llama, Claude) -> next-token prediction.** The model reads tokens left to right and predicts the next token at every position, using only the tokens that preceded it. The causal mask blocks attention to future positions, which means the model cannot cheat by looking ahead. This is what makes the model useful for generation: at inference time, it predicts one token at a time using the same mechanism it learned during pretraining. **Encoder-decoder (T5, FLAN-T5) -> span-corruption denoising.** Random SPANS (not single tokens) in the input are replaced with sentinel tokens like `<extra_id_0>`. The encoder sees the corrupted sequence with bidirectional attention; the decoder generates the original spans one sentinel at a time. This combines bidirectional understanding (in the encoder) with autoregressive generation (in the decoder). **Encoder-decoder cross-modal (Whisper) -> supervised audio to text seq2seq.** The encoder reads a mel-spectrogram of audio with bidirectional attention; the decoder generates the corresponding text transcription autoregressively. Unlike the others, this is supervised pretraining on labeled audio-text pairs, not self-supervised on raw text. The architecture is encoder-decoder because the input modality (audio) is structurally different from the output modality (text). **Why the pairings are not arbitrary.** Each objective requires a specific attention pattern. MLM needs bidirectional context to fill in the blank from both sides. Next-token prediction needs a causal mask to avoid cheating. Span corruption needs an encoder for bidirectional reading and a decoder for autoregressive generation. Cross-modal seq2seq needs separate encoder and decoder because the modalities differ.

**Pairings.** - Encoder-only (BERT, ModernBERT) -> **masked language modeling**. Random 15% of input tokens are masked, the model uses bidirectional attention to predict them. - Decoder-only (GPT, Llama, Claude) -> **causal next-token prediction**. Every position predicts the next token using only past context, enforced by a triangular causal mask. - Encoder-decoder (T5, FLAN-T5) -> **span-corruption denoising**. Random spans (typically 15% of tokens in 3-token spans) are replaced with sentinel tokens, the decoder generates the original spans autoregressively. - Encoder-decoder cross-modal (Whisper) -> **supervised audio to text seq2seq**. Encoder reads mel-spectrogram with bidirectional attention, decoder generates text transcription token by token. **Why each pairing is canonical.** **Encoder-only + MLM.** The encoder's lack of a causal mask is the key feature; every token can attend to every other token in both directions. This bidirectional context is what lets MLM work. A token in position 5 can use information from positions 1-4 (left context) and 6-100 (right context) to predict what should be at position 5. Try MLM with a causal mask and the model could only use left context, which wastes half the signal. Try next-token prediction with bidirectional attention and the model would trivially see the answer (the next token is in its receptive field), so the objective becomes useless. The mask and objective have to match. **Decoder-only + next-token.** The causal mask makes next-token prediction self-supervised at scale: every training token is a prediction target, every previous token is context. The mask prevents the trivial cheat of attending to the future. This is the most data-efficient text objective because it produces L training signals from an L-token sequence (versus ~0.15L for MLM with 15% masking).