Modern LLM fine-tuning defaults to causal LM or masked LM, pick one and say why

MCQ·Easy·4.0 · 0·~1 min·Asked atNeo4j Vellum·Relevant atDatabricks

Attempt it

There are two classic language modeling objectives. **Causal LM** predicts the next token given everything before it, left to right. The attention mask in the model blocks each position from attending to any position after itself. Decoder-only models like Llama, Mistral, Qwen, DeepSeek, and the GPT family use this. **Masked LM** randomly hides some tokens in a sequence and asks the model to predict them from the unhidden context on both sides. The attention mask is bidirectional. Encoder-only models like BERT and RoBERTa use this. Modern instruction fine-tuning targets the deployed generative LLMs, which are all decoder-only. Those models have a causal attention mask baked into the architecture. The fine-tuning objective must match what the model was pretrained with, so it must be causal LM. Using MLM on a decoder-only model would break the assumption the architecture relies on. The model has never seen bidirectional context during pretraining, and forcing the fine-tuning objective to use bidirectional context would push the weights to optimise something the architecture cannot represent cleanly. The result is degraded base capability and no real task gain. The other distractor options describe hybrid objectives that no published instruction-tuned LLM uses, because they break either the architectural mask or the objective continuity.

**Causal LM** is the answer because every dominant deployed LLM (Llama 4, Mistral, Qwen 3.5, DeepSeek V4, GPT 5.5, Gemini 3.1) is a decoder-only transformer pretrained on left to right next-token prediction. The key constraint is architectural. A decoder-only transformer has a causal self-attention mask, meaning each token at position t can only attend to positions less than or equal to t. This mask is hardcoded into the attention computation: $$M_{ij} = \begin{cases} 0 & i \geq j \\ -\infty & i < j \end{cases}$$ Masked LM requires bidirectional attention, where every position attends to every other position. That requires a different mask, a different pretraining objective, and a different family of models (encoder-only like BERT or encoder-decoder like T5). The fine-tuning objective must match the pretraining objective. If pretraining was causal next-token prediction, fine-tuning must also be causal next-token prediction. Trying to apply MLM to a decoder-only model forces it to optimise an objective its mask cannot satisfy cleanly. The model unlearns some of its pretrained capability and never reaches the new objective's intended quality. The hybrid options (50/50 alternation, per-segment mix) are not used by any published recipe because they break either the mask assumption or the objective continuity. Modern instruction SFT is plain causal next-token prediction on the response tokens, masked out of the loss for prompt tokens, identical in form to pretraining.

TL;DR

Modern instruction tuning uses causal LM because all major deployed LLMs are decoder-only with a causal attention mask, and the fine-tuning objective must match the pretraining objective.

Memory aid

Easy to grasp

Imagine two different language games. In game one you read a sentence with random words covered up and guess each missing word by looking at words on both sides. In game two you read a sentence from left to right and try to predict the next word using only what came before. Both are good language exercises, but they need different rules. The first game uses a referee who lets you peek both ways, the second uses a referee who blocks you from looking ahead. The big modern chatbots like Llama and Mistral and DeepSeek were trained by playing the second game from birth. Their internal referee is built for left to right only. If you try to teach them the first game later, the referee gets confused and the lesson goes nowhere. So fine-tuning has to play the same game the model was raised on.

Key concepts

Causal language modeling Masked language modeling Decoder-only architecture Pretraining objective

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Choosing the fine-tuning objective is a question about architectural compatibility, not preference. The answer is determined by what kind of model you are fine-tuning, and for the entire current generation of deployed LLMs that answer is causal language modeling.

The question lays out four options. Two are real objectives from different model families, and two are hybrid recipes that no published frontier model actually uses. The reasoning that picks the right one is the same reasoning that explains why MLM is the wrong choice and why the hybrids are not real recipes: the pretraining objective and the architectural mask are coupled, and fine-tuning has to respect that coupling.

This deep dive walks through the two classical objectives, the architectures they belong to, why the objective and mask cannot be mixed and matched, the consequences of trying to fine-tune with the wrong objective, and why the hybrid distractor options are mechanical impossibilities for a decoder-only model. By the end the causal LM choice should look not just correct but inevitable.

The two classical objectives and the architectures they belong to

Two language modeling objectives dominated the early transformer era and shaped two distinct model families.

Causal language modeling

Causal LM, also called autoregressive LM, predicts the next token in a sequence given all preceding tokens. The loss at each position is the negative log probability of the actual next token under the model's predicted distribution. Training proceeds left to right; the model is forbidden from seeing tokens to the right of the current position.

This objective is enforced architecturally by a causal attention mask:

M_{ij} = \begin{cases} 0 & i \geq j \\ -\infty & i < j \end{cases}

The mask is added to the pre-softmax attention scores. The negative infinity values become zero after softmax, so position i attends only to positions less than or equal to i. The mask is hardcoded into the model's attention layers, not a data preprocessing choice.

Decoder-only transformers use this objective and this mask: the GPT family, Llama 4, Mistral, Qwen 3.5, DeepSeek V4, Gemma 4, every major open-weight model.

Masked language modeling

MLM randomly masks some fraction of the input tokens, replaces them with a special MASK token, and asks the model to predict the original tokens from the surrounding bidirectional context. The loss is computed only at masked positions.

MLM requires bidirectional attention. Every position can attend to every other position; there is no causal mask. The model sees the entire sequence, including positions to the right of the masked tokens, when making predictions.

Encoder-only transformers use this objective: BERT, RoBERTa, DeBERTa, ALBERT. These models are used for embeddings, classification, named entity recognition, and other tasks where the entire input is available at inference and bidirectional context helps.

Why the two cannot be mixed in one model

The attention mask is the gate. A model with a causal mask cannot do MLM cleanly, because predicting a masked token at position i would require attending to tokens at positions greater than i, which the mask forbids. A model with bidirectional attention can technically do CLM, but doing so wastes the bidirectional capacity and tends to produce models that are worse generators than purpose-built decoder-only ones.

Why the fine-tuning objective must match pretraining

Why the hybrid options are not real recipes

How loss masking and the causal objective work together

Putting it together

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Llama 4 and Mistral 7B fine-tuning recipes in Hugging Face TRL use causal language modeling with response-only loss masking on chat formatted data.
Qwen 3.5 and DeepSeek V4 distillation pipelines run plain CLM next-token prediction over chat templated sequences with prompt tokens masked from the loss.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is the causal attention mask considered a hardcoded architectural feature rather than a data property?

Explain that the mask is applied inside the attention computation at every layer and is part of the forward pass; the model cannot see future tokens regardless of what the data looks like, so the mask is structural.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating MLM and CLM as interchangeable training objectives. They are not; the attention mask in the base architecture determines which one is structurally compatible and trying to fine-tune with the wrong objective degrades the model.

60 second bullets to scan on the way to the call.

Which pretraining objective decoder-only LLMs use
Why the attention mask determines the objective

Primary sources. Browse if you want the original framing.