Why can BERT use bidirectional attention but decoder only LLMs like GPT cannot?
Training inference consistency. BERT's MLM objective is bidirectional at both train and inference.
Imagine two kinds of word puzzles. The first is a crossword: a few letters are blanked out and you can see every other letter on the grid, left, right, above, below, to fill in the missing ones. That's BERT: it learns by filling blanks while seeing the whole sentence at once. The second is reading a mystery one line at a time, with a piece of paper covering everything below the line you're on, and guessing the next word before sliding the paper down. That's GPT: it only ever sees what came before. Compare the two: BERT works because the test (a sentence with a few blanks) looks just like training. GPT works because the test (write the next word) also looks just like training. Mixing them, training crossword style but testing mystery style, would be like studying with the answer key open and then taking the real test in a quiet room.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
The bidirectional vs causal choice looks like an architecture question but is really a training inference consistency question. The architecture follows from what information is available when the model is actually used.
BERT, GPT, T5, and UL2 all use the same underlying transformer block, the only structural difference is the attention mask. What separates them is how the pretraining objective lines up with the inference modality. This deep dive walks the two pure cases (BERT and GPT), explains why mixing them fails empirically, and surveys the hybrid architectures that try to harvest both.
Mental model: match the attention pattern at training to what the model can see at inference. Break the symmetry and generation collapses.
BERT: bidirectional at both train and inference
The MLM objective
BERT is pretrained on masked language modeling. About 15% of input tokens are replaced with [MASK], and the model predicts the original token at each masked position from the surrounding bidirectional context.
- Self-attention has no causal mask: every position attends to every other.
- The
[MASK]token at masked positions prevents trivial leakage at those specific positions. - Loss is computed only on the masked positions.
The inference modality
BERT's downstream use cases all share the same shape: the input is fully observed at once, and the model produces either a sequence level vector (classification), per token tags (NER), or a span (extractive QA).
- Classification: pool the
[CLS]token's representation. - NER / tagging: per token softmax over label set.
- Retrieval: pool to a single vector, compute cosine similarity.
Consistency check
In both train and inference modes, every position has access to its full left and right context. The architecture lines up cleanly with how the model is used.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
| Model | Pretraining | Attention | Inference modality |
|---|---|---|---|
| BERT | Masked LM | Bidirectional | Encode full input once |
| GPT / Llama | Next token prediction | Causal | Autoregressive generation |
| T5 | Span corruption | Bi (enc) + Causal (dec) | Seq2seq generation |
| UL2 | Mixed denoisers | Bi + Prefix-LM + Causal | Both encode and generate |
Real products, models, and research that use this idea.
- BERT: bidirectional MLM training, bidirectional encoding at inference. Used for classification, NER, QA via fine tuning.
- GPT-2/3/4 and Llama/Mistral: causal next token prediction at training, autoregressive generation at inference.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat goes wrong if you fine tune BERT for generation with autoregressive decoding?
Two problems: (1) BERT's pretraining doesn't optimize for next token quality, so its representations aren't well calibrated for generation; (2) bidirectional training relied on future context, but at autoregressive inference that context doesn't exist. You CAN do it (some work on BERT2BERT) but quality is worse than purpose built decoder LLMs of equal size.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Treating it as a technical impossibility rather than a training inference consistency requirement. Bidirectional attention is mechanically fine; the issue is what's available at generation time.
60 second bullets to scan on the way to the call.
Training inference consistency principle
BERT's MLM objective and bidirectional inference modality
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.