Why can BERT use bidirectional attention but decoder only LLMs like GPT cannot?

**Bidirectional** attention and causal attention are mechanically almost identical: they differ only in whether a mask is applied. So the question isn't 'can a decoder LLM do bidirectional attention?' (it technically could) but 'would it work?' (no). The answer comes down to training inference consistency: the model must learn from contexts that match what it will see at inference. BERT trains on masked language modeling. About 15% of tokens are replaced with `[MASK]`, and the model predicts each masked token using full bidirectional context. At inference, BERT is used for classification, tagging, retrieval: tasks where the whole input is available at once. Bidirectional at train, bidirectional at inference. GPT trains on next token prediction with a causal mask, so position t only sees positions 1..t. At inference, generation works the same way: position t still only has past tokens, because t+1 hasn't been generated yet. Train bidirectionally and the model would rely on future context that simply isn't there at generation time.

The real question isn't whether decoder LLMs *can* use bidirectional attention, they mechanically can, it's about **training inference consistency**. BERT trains on masked language modeling, predicting masked tokens from full bidirectional context. At inference for downstream tasks like classification or retrieval, BERT sees the full input at once. Bidirectional at train, bidirectional at inference. Consistent. GPT trains on next token prediction with a causal mask, and at inference does autoregressive generation. At generation step t, only tokens 1..t-1 actually exist: there is no t+1 to attend to. So causal at train, causal at inference. Also consistent. If you trained GPT with bidirectional attention, the model would learn to rely on future context: but at generation time that context doesn't exist. The train inference mismatch produces degraded or nonsensical outputs. This is the actual constraint. A few tradeoffs worth naming: bidirectional attention costs the same as causal (just no mask), so it's not a compute argument. And you can mix them, T5 uses a bidirectional encoder with a causal decoder, and UL2 trains with a mix of bidirectional and prefix-LM objectives. But the 2023+ consensus has been that pure decoder only with causal attention scales better and is what frontier LLMs use.