There are two classic language modeling objectives.
**Causal LM** predicts the next token given everything before it, left to right. The attention mask in the model blocks each position from attending to any position after itself. Decoder-only models like Llama, Mistral, Qwen, DeepSeek, and the GPT family use this.
**Masked LM** randomly hides some tokens in a sequence and asks the model to predict them from the unhidden context on both sides. The attention mask is bidirectional. Encoder-only models like BERT and RoBERTa use this.
Modern instruction fine-tuning targets the deployed generative LLMs, which are all decoder-only. Those models have a causal attention mask baked into the architecture. The fine-tuning objective must match what the model was pretrained with, so it must be causal LM.
Using MLM on a decoder-only model would break the assumption the architecture relies on. The model has never seen bidirectional context during pretraining, and forcing the fine-tuning objective to use bidirectional context would push the weights to optimise something the architecture cannot represent cleanly. The result is degraded base capability and no real task gain.
The other distractor options describe hybrid objectives that no published instruction-tuned LLM uses, because they break either the architectural mask or the objective continuity.
**Causal LM** is the answer because every dominant deployed LLM (Llama 4, Mistral, Qwen 3.5, DeepSeek V4, GPT 5.5, Gemini 3.1) is a decoder-only transformer pretrained on left to right next-token prediction.
The key constraint is architectural. A decoder-only transformer has a causal self-attention mask, meaning each token at position t can only attend to positions less than or equal to t. This mask is hardcoded into the attention computation:
$$M_{ij} = \begin{cases} 0 & i \geq j \\ -\infty & i < j \end{cases}$$
Masked LM requires bidirectional attention, where every position attends to every other position. That requires a different mask, a different pretraining objective, and a different family of models (encoder-only like BERT or encoder-decoder like T5).
The fine-tuning objective must match the pretraining objective. If pretraining was causal next-token prediction, fine-tuning must also be causal next-token prediction. Trying to apply MLM to a decoder-only model forces it to optimise an objective its mask cannot satisfy cleanly. The model unlearns some of its pretrained capability and never reaches the new objective's intended quality.
The hybrid options (50/50 alternation, per-segment mix) are not used by any published recipe because they break either the mask assumption or the objective continuity. Modern instruction SFT is plain causal next-token prediction on the response tokens, masked out of the loss for prompt tokens, identical in form to pretraining.
Quick reference
TL;DR
Modern instruction tuning uses causal LM because all major deployed LLMs are decoder-only with a causal attention mask, and the fine-tuning objective must match the pretraining objective.
Imagine two different language games. In game one you read a sentence with random words covered up and guess each missing word by looking at words on both sides. In game two you read a sentence from left to right and try to predict the next word using only what came before. Both are good language exercises, but they need different rules. The first game uses a referee who lets you peek both ways, the second uses a referee who blocks you from looking ahead. The big modern chatbots like Llama and Mistral and DeepSeek were trained by playing the second game from birth. Their internal referee is built for left to right only. If you try to teach them the first game later, the referee gets confused and the lesson goes nowhere. So fine-tuning has to play the same game the model was raised on.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Fine-tuning a BERT-family encoder model for classification or embeddings Encoder models use bidirectional attention and were pretrained with MLM, so CLM is the wrong objective and would not produce useful weights.
Fine-tuning an encoder-decoder model like T5 or BART for sequence to sequence tasks These models have a separate encoder and decoder with different objectives, and the standard fine-tuning recipe uses the original span corruption or sequence to sequence loss, not pure CLM.
Training an embedding model where bidirectional context matters Embedding models benefit from bidirectional attention and an MLM or contrastive objective, not the causal forward-only attention pattern.
Specialised research on hybrid architectures that mix encoder and decoder layers Custom architectures may require custom objectives that do not map onto the pure CLM or MLM choice; these are research settings, not standard fine-tuning.
Quick scan · 2–4 min
Everything important, quickly.
4 min: name causal LM, point at the decoder-only architecture and causal mask, contrast with the MLM and encoder-only family, explain why fine-tuning must match pretraining, and dispatch the two hybrid distractors.
Real-world examples
Real products, models, and research that use this idea.
1Llama 4 and Mistral 7B fine-tuning recipes in Hugging Face TRL use causal language modeling with response-only loss masking on chat formatted data.
2Qwen 3.5 and DeepSeek V4 distillation pipelines run plain CLM next-token prediction over chat templated sequences with prompt tokens masked from the loss.
3BERT-family encoder models like RoBERTa and DeBERTa are fine-tuned with MLM continued pretraining or with task-specific classifier heads, never with CLM, because the bidirectional attention mask is incompatible.
4T5 and its descendants use a span corruption objective that is closer in spirit to MLM than to CLM, reflecting their encoder-decoder architecture and the bidirectional encoder side.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy is the causal attention mask considered a hardcoded architectural feature rather than a data property?
Sign in or go Pro to see the approach for this follow-up.
QWhat happens to a decoder-only model if you continue pretraining it with an MLM objective?
Sign in or go Pro to see the approach for this follow-up.
QWhy do encoder-decoder models like T5 use a different pretraining objective than either pure CLM or pure MLM?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating MLM and CLM as interchangeable training objectives. They are not; the attention mask in the base architecture determines which one is structurally compatible and trying to fine-tune with the wrong objective degrades the model.