**Bidirectional** attention and causal attention are mechanically almost identical: they differ only in whether a mask is applied. So the question isn't 'can a decoder LLM do bidirectional attention?' (it technically could) but 'would it work?' (no).
The answer comes down to training inference consistency: the model must learn from contexts that match what it will see at inference.
BERT trains on masked language modeling. About 15% of tokens are replaced with `[MASK]`, and the model predicts each masked token using full bidirectional context. At inference, BERT is used for classification, tagging, retrieval: tasks where the whole input is available at once. Bidirectional at train, bidirectional at inference.
GPT trains on next token prediction with a causal mask, so position t only sees positions 1..t. At inference, generation works the same way: position t still only has past tokens, because t+1 hasn't been generated yet. Train bidirectionally and the model would rely on future context that simply isn't there at generation time.
The real question isn't whether decoder LLMs *can* use bidirectional attention, they mechanically can, it's about **training inference consistency**.
BERT trains on masked language modeling, predicting masked tokens from full bidirectional context. At inference for downstream tasks like classification or retrieval, BERT sees the full input at once. Bidirectional at train, bidirectional at inference. Consistent.
GPT trains on next token prediction with a causal mask, and at inference does autoregressive generation. At generation step t, only tokens 1..t-1 actually exist: there is no t+1 to attend to. So causal at train, causal at inference. Also consistent.
If you trained GPT with bidirectional attention, the model would learn to rely on future context: but at generation time that context doesn't exist. The train inference mismatch produces degraded or nonsensical outputs. This is the actual constraint.
A few tradeoffs worth naming: bidirectional attention costs the same as causal (just no mask), so it's not a compute argument. And you can mix them, T5 uses a bidirectional encoder with a causal decoder, and UL2 trains with a mix of bidirectional and prefix-LM objectives. But the 2023+ consensus has been that pure decoder only with causal attention scales better and is what frontier LLMs use.
Quick reference
TL;DR
Training inference consistency. BERT's MLM objective is bidirectional at both train and inference.
Imagine two kinds of word puzzles. The first is a crossword: a few letters are blanked out and you can see every other letter on the grid, left, right, above, below, to fill in the missing ones. That's BERT: it learns by filling blanks while seeing the whole sentence at once. The second is reading a mystery one line at a time, with a piece of paper covering everything below the line you're on, and guessing the next word before sliding the paper down. That's GPT: it only ever sees what came before. Compare the two: BERT works because the test (a sentence with a few blanks) looks just like training. GPT works because the test (write the next word) also looks just like training. Mixing them, training crossword style but testing mystery style, would be like studying with the answer key open and then taking the real test in a quiet room.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Any generative task, story writing, chat completion, code synthesis, summarization output Bidirectional training relies on right side context that doesn't exist during autoregressive generation; the train inference mismatch collapses generation quality.
Chain of thought or step by step reasoning that unfolds left to right Each reasoning step conditions on the previous steps already being committed; bidirectional access would let the model peek at later steps and short circuit the chain.
Streaming inference where tokens are produced one at a time with low latency Bidirectional attention requires the full sequence before any output, blocking incremental decode and breaking real time chat interfaces.
Training with KV-cache-friendly serving as a requirement KV caching depends on the invariant that earlier positions never attend to newer ones; bidirectional attention violates that, forcing a full re-compute on every new token.
Quick scan · 2–4 min
Everything important, quickly.
Walk through MLM vs next token prediction objectives, the inference modality of each, why mixing breaks generation, hybrid approaches (T5, prefix-LM, UL2), and the modern decoder only consensus.
Side-by-side
Model
Pretraining
Attention
Inference modality
BERT
Masked LM
Bidirectional
Encode full input once
GPT / Llama
Next token prediction
Causal
Autoregressive generation
T5
Span corruption
Bi (enc) + Causal (dec)
Seq2seq generation
UL2
Mixed denoisers
Bi + Prefix-LM + Causal
Both encode and generate
Real-world examples
Real products, models, and research that use this idea.
1BERT: bidirectional MLM training, bidirectional encoding at inference. Used for classification, NER, QA via fine tuning.
2GPT-2/3/4 and Llama/Mistral: causal next token prediction at training, autoregressive generation at inference.
3T5: bidirectional encoder + causal decoder; gets the best of both for seq2seq tasks like translation/summarization.
4UL2: mixes MLM, prefix-LM, and span corruption objectives in a single model.
5ELECTRA: alternative to BERT-style MLM using a discriminator: still bidirectional encoding.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat goes wrong if you fine tune BERT for generation with autoregressive decoding?
Sign in or go Pro to see the approach for this follow-up.
QWhy is T5 an encoder decoder rather than decoder only?
Sign in or go Pro to see the approach for this follow-up.
QDoes prefix-LM give you 'the best of both worlds'?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Treating it as a technical impossibility rather than a training inference consistency requirement. Bidirectional attention is mechanically fine; the issue is what's available at generation time.