Which modern LLM architecture uses cross-attention?

Cross-attention is one of two flavors of attention in transformers, and which flavor a model uses depends entirely on the architectural family. Self-attention is when Q, K, and V are all produced from the same input sequence. Every modern transformer layer has at least one self-attention block. Cross-attention is when Q comes from one sequence and K and V come from a different sequence. The canonical use case is translation: an encoder reads the source language and a decoder generates the target language while attending back to the encoder's output. The three architectural families to remember are: - Encoder decoder: T5, BART, the original transformer, Whisper. Has both encoder and decoder, and the decoder uses cross-attention. - Encoder-only: BERT, RoBERTa, DeBERTa. Bidirectional self-attention. No decoder, no cross-attention. - Decoder-only: GPT family, Llama, Mistral, Claude. Causal self-attention. No encoder, no cross-attention. So the answer is T5 and BART. The famous chat LLMs are decoder-only and do not use cross-attention at all.

Cross-attention lives in the decoder of encoder decoder models, which means T5 and BART among the options here. In an encoder decoder model, the encoder processes the source sequence into a stack of hidden states. The decoder then has cross-attention layers where Q is projected from the decoder's own residual stream and K, V are projected from the frozen encoder output. That cross-attention is what lets the decoder ground its generation in the source. The other options are wrong for a specific reason each. Llama is decoder-only: no encoder at all, so no cross-attention. BERT is encoder-only, all bidirectional self-attention. And the claim that 'all modern transformers use both' is just false: every modern chat LLM since GPT-3 has been decoder-only, with no cross-attention anywhere in the stack. One trade off worth naming: the field largely abandoned cross-attention for general LLMs because decoder-only architectures are simpler to scale and serve. You prepend everything into one sequence and let causal self-attention handle context: no separate encoder pass, no two tower architecture. Cross-attention is still alive in multimodal models like Flamingo and in speech to text encoder decoders like Whisper, but for pure-text chat models, it's gone.