Modern LLM fine-tuning defaults to causal LM or masked LM, pick one and say why

MCQ·Easy·4.0 · 0·~1 min

Attempt it

There are two classic language modeling objectives. **Causal LM** predicts the next token given everything before it, left to right. The attention mask in the model blocks each position from attending to any position after itself. Decoder-only models like Llama, Mistral, Qwen, DeepSeek, and the GPT family use this. **Masked LM** randomly hides some tokens in a sequence and asks the model to predict them from the unhidden context on both sides. The attention mask is bidirectional. Encoder-only models like BERT and RoBERTa use this. Modern instruction fine-tuning targets the deployed generative LLMs, which are all decoder-only. Those models have a causal attention mask baked into the architecture. The fine-tuning objective must match what the model was pretrained with, so it must be causal LM. Using MLM on a decoder-only model would break the assumption the architecture relies on. The model has never seen bidirectional context during pretraining, and forcing the fine-tuning objective to use bidirectional context would push the weights to optimise something the architecture cannot represent cleanly. The result is degraded base capability and no real task gain. The other distractor options describe hybrid objectives that no published instruction-tuned LLM uses, because they break either the architectural mask or the objective continuity.

**Causal LM** is the answer because every dominant deployed LLM (Llama 4, Mistral, Qwen 3.5, DeepSeek V4, GPT 5.5, Gemini 3.1) is a decoder-only transformer pretrained on left to right next-token prediction. The key constraint is architectural. A decoder-only transformer has a causal self-attention mask, meaning each token at position t can only attend to positions less than or equal to t. This mask is hardcoded into the attention computation: $$M_{ij} = \begin{cases} 0 & i \geq j \\ -\infty & i < j \end{cases}$$ Masked LM requires bidirectional attention, where every position attends to every other position. That requires a different mask, a different pretraining objective, and a different family of models (encoder-only like BERT or encoder-decoder like T5). The fine-tuning objective must match the pretraining objective. If pretraining was causal next-token prediction, fine-tuning must also be causal next-token prediction. Trying to apply MLM to a decoder-only model forces it to optimise an objective its mask cannot satisfy cleanly. The model unlearns some of its pretrained capability and never reaches the new objective's intended quality. The hybrid options (50/50 alternation, per-segment mix) are not used by any published recipe because they break either the mask assumption or the objective continuity. Modern instruction SFT is plain causal next-token prediction on the response tokens, masked out of the loss for prompt tokens, identical in form to pretraining.

Quick reference

TL;DR

Modern instruction tuning uses causal LM because all major deployed LLMs are decoder-only with a causal attention mask, and the fine-tuning objective must match the pretraining objective.

Memory aid

Easy to grasp

Imagine two different language games. In game one you read a sentence with random words covered up and guess each missing word by looking at words on both sides. In game two you read a sentence from left to right and try to predict the next word using only what came before. Both are good language exercises, but they need different rules. The first game uses a referee who lets you peek both ways, the second uses a referee who blocks you from looking ahead. The big modern chatbots like Llama and Mistral and DeepSeek were trained by playing the second game from birth. Their internal referee is built for left to right only. If you try to teach them the first game later, the referee gets confused and the lesson goes nowhere. So fine-tuning has to play the same game the model was raised on.

Key concepts

Causal language modeling Masked language modeling Decoder-only architecture Pretraining objective

Detailed answer & concept explanation

~9 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 min: name causal LM, point at the decoder-only architecture and causal mask, contrast with the MLM and encoder-only family, explain why fine-tuning must match pretraining, and dispatch the two hybrid distractors.

Real-world examples

Real products, models, and research that use this idea.

Llama 4 and Mistral 7B fine-tuning recipes in Hugging Face TRL use causal language modeling with response-only loss masking on chat formatted data.
Qwen 3.5 and DeepSeek V4 distillation pipelines run plain CLM next-token prediction over chat templated sequences with prompt tokens masked from the loss.
BERT-family encoder models like RoBERTa and DeBERTa are fine-tuned with MLM continued pretraining or with task-specific classifier heads, never with CLM, because the bidirectional attention mask is incompatible.
T5 and its descendants use a span corruption objective that is closer in spirit to MLM than to CLM, reflecting their encoder-decoder architecture and the bidirectional encoder side.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is the causal attention mask considered a hardcoded architectural feature rather than a data property?

QWhat happens to a decoder-only model if you continue pretraining it with an MLM objective?

QWhy do encoder-decoder models like T5 use a different pretraining objective than either pure CLM or pure MLM?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Treating MLM and CLM as interchangeable training objectives. They are not; the attention mask in the base architecture determines which one is structurally compatible and trying to fine-tune with the wrong objective degrades the model.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.