Name a task where an encoder-decoder transformer still beats a decoder-only LLM

Short answer·Easy·4.0 · 0·~3 min·Asked atNVIDIA·Relevant atGoogle Mistral AI

Attempt it

Pick one production task where an encoder-decoder transformer is still the right choice, and explain what specifically makes encoder-decoder a better fit than a decoder-only LLM.

Free · 2 AI evals / day

Encoder-decoder transformers split the work into two stacks. The **encoder** reads the input once and builds a representation. The **decoder** generates the output one token at a time while cross-attending to that fixed encoder representation. The classic fit is speech-to-text. Whisper and NVIDIA NeMo Canary use this shape. The encoder ingests a mel-spectrogram (a picture of sound frequencies) of the audio in one pass and produces a sequence of hidden states. The decoder then emits text tokens one by one, cross-attending to that cached encoder output at every step. The audio is never re-encoded. The same representation drives the entire transcript. A decoder-only LLM can transcribe audio when paired with an audio tokenizer. But the audio tokens and every new text token all run through the whole large model, while Whisper's decoder is small. The cost per second of audio is higher. The latency is also worse on a fixed-format task like transcription, where the input and the output are both bounded. Translation is the second classic fit. NLLB-200 and M2M-100 serve dozens of language pairs over a single shared encoder. Bidirectional context on the source side helps disambiguate words before the decoder commits to a target word.

**Speech-to-text** is the classic fit for encoder-decoder. The encoder ingests the mel-spectrogram once and produces a sequence of hidden states. The decoder autoregressively emits text while cross-attending to that cached encoding. The audio representation is built once and reused at every decoder step at zero recompute cost. A decoder-only LLM can technically do transcription (with an audio tokenizer like the one in GPT-4o-audio). But the audio tokens, many times longer than the transcript, run through the full model in one causal sequence, and so does every generated text token. Whisper's decoder can be small because the encoder does the heavy work. The cost per minute of audio is meaningfully higher and the latency story is worse. On a bounded-input / bounded-output task like transcription, the encoder-decoder split wins on both axes. Whisper-large-v3, NVIDIA NeMo Canary, and Distil-Whisper are all production examples. Machine translation is the second strong fit. NLLB-200, M2M-100, and mBART use a shared encoder across many language pairs. Bidirectional context on the source side helps disambiguate before the decoder commits. Decoder-only LLMs are very good at translation now too. But at high request volume for fixed-format pairs, the encoder-decoder is still cheaper per token. **The structural property that drives all of this:** encoder-decoder wins when the input modality differs from the output, the input is bounded, the output is bounded, and the same input might drive multiple generations (re-decoding with different sampling temperatures, beam search, n-best candidates). The encoder pays once; the decoder reuses.

TL;DR

Speech-to-text. Whisper-style encoder-decoders run the encoder once on the audio, then the decoder cross-attends to that cached encoding at every step: far cheaper than a decoder-only LLM ingesting audio tokens

Memory aid

Easy to grasp

Imagine you have a long voice message and you want it written down. An encoder-decoder model is like having one specialist who listens to the whole recording from start to finish and writes a summary on a whiteboard, then a writer who looks at that whiteboard while typing the transcript word by word. The whiteboard is written once and read many times. A decoder-only LLM is like one very large expert who both listens and types, using their whole effort for every single word typed. It works, but each word costs much more than it does for the small writer. That's the core reason Whisper and other speech models still beat decoder-only LLMs on cost and latency for transcription. This holds even though decoder-only LLMs are technically capable of doing it.

Key concepts

Attention

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

The transformer comes in two main flavors: the encoder-decoder (the original 2017 'Attention Is All You Need' shape, built for translation) and the decoder-only (GPT, used for left to right generation). The field has since consolidated around decoder-only LLMs, to the point where new practitioners often assume encoder-decoder is a legacy shape. It is not.

Encoder-decoder owns a specific corner of the workload space: tasks where the input is bounded and reused across many decoder steps, where the input modality differs from the output modality, and where unit economics at scale matter more than open-ended generation quality. Speech-to-text is the textbook example, machine translation is the strong second, and bounded structured extraction is the quiet third.

What the encoder-decoder split does

An encoder-decoder transformer processes input in two distinct phases. The encoder stack runs once over the input sequence and produces a sequence of hidden states with shape [encoder_seq_len, d_model]. These hidden states encode bidirectional context (every position can attend to every other position during encoding). They are then frozen for the duration of the query.

The decoder stack generates output autoregressively, one token at a time. Each decoder layer has two attention sub-blocks. Self-attention lets the decoder attend over the tokens it has already generated (masked to be causal). Cross-attention lets the decoder attend over the encoder's hidden states. The encoder hidden states are the K and V, and the decoder's own hidden state at the current step is the Q.

The key property: the encoder runs once, and the decoder runs many times. The encoder hidden states are computed once at the start of decoding and reused at every later decoder step at zero recompute cost. For a 30-second audio clip that produces a 100-token transcript, the encoder runs once and the decoder runs 100 times against the same cached encoder hidden states.

A decoder-only transformer has no such split. The input and the output sit in one sequence, separated by special tokens. Every decoder step self-attends over the entire prefix, which includes the original input. The KV cache of the input is reused (so the input is not re-encoded), and each step attends over that cached prefix, just as cross-attention reads every encoder position. The difference is that the input passes through the same causal stack as the output, with no bidirectional encoder and no way to make the generating part smaller.

Speech-to-text: the classic fit

Translation: the strong second fit

The general structural rule

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Property	Encoder-decoder	Decoder-only LLM
Input encoding cost	Once per query	Once per query (but in same sequence)
Per-decode-step attention	Self-attn over text + cross-attn over cached encoder K/V	Self-attn over full audio + text prefix
Cost-per-audio-second	Low (Whisper-class)	Multiples higher (audio-LLM)
Best fit	Bounded input + bounded output (transcription, translation, extraction)	Open-ended generation, reasoning, multi-turn chat

Real products, models, and research that use this idea.

Whisper-large-v3 is the workhorse open-weight speech-to-text encoder-decoder; it powers podcast transcription pipelines, call-center analytics, and meeting-summary products.
Distil-Whisper is a distilled encoder-decoder variant that runs at roughly 6x the speed of Whisper-large for production transcription at scale.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is the audio-token sequence so much longer than the text-token sequence for the same content?

Speech tokenizers operate at frame rates of 25-50 Hz on the audio side, while text tokenizers compress to roughly 3-4 characters per token. A 30-second clip of speech becomes ~1500 audio tokens and ~75 text tokens. The 20x ratio is what makes encoder-decoder attractive for transcription.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Claiming decoder-only models have fully replaced encoder-decoders. They have not. Speech to text, large-batch translation, and bounded-output structured extraction still favor encoder-decoder on cost per token and latency.

60 second bullets to scan on the way to the call.

Cross-attention in the decoder and how it reuses the encoder's cached hidden states
Why audio token sequences are much longer than the resulting text sequences

Primary sources. Browse if you want the original framing.