Name a 2026 task where an encoder-decoder transformer still beats a decoder-only LLM

Short answer·Easy·4.0 · 0·~3 min·Asked atNVIDIA Tcs·Relevant atGoogle Mistral AI

Attempt it

Pick one production task where an encoder-decoder transformer is still the right choice in 2026, and explain what specifically makes encoder-decoder a better fit than a decoder-only LLM.

Free · 2 AI evals / day

Encoder-decoder transformers split the work into two stacks. The **encoder** reads the input once and builds a representation. The **decoder** generates the output one token at a time while cross-attending to that fixed encoder representation. The canonical 2026 fit is speech-to-text. Whisper, NVIDIA NeMo Canary, and Distil-Whisper all use this shape. The encoder ingests a mel-spectrogram of the audio in one pass and produces a sequence of hidden states. The decoder then emits text tokens one by one, cross-attending to that cached encoder output at every step. The audio is never re-encoded; the same representation drives the entire transcript. A decoder-only LLM in 2026 (GPT-5.5, Llama 3.1 70B, Claude) can technically transcribe audio when paired with an audio tokenizer, but each decoder step has to attend over the full audio token stream, which is much longer than the corresponding text. The cost per second of audio is higher and the latency is worse on a fixed-format task like transcription where the input is bounded and the output is bounded. Translation is the second classic fit. NLLB-200 and M2M-100 serve dozens of language pairs over a single shared encoder. Bidirectional context on the source side helps disambiguate words before the decoder commits to a target word.

**Speech-to-text** is the canonical 2026 fit for encoder-decoder. The encoder ingests the mel-spectrogram once and produces a sequence of hidden states; the decoder autoregressively emits text while cross-attending to that cached encoding. The audio representation is built once and reused at every decoder step at zero recompute cost. A decoder-only LLM can technically do transcription in 2026 (with an audio tokenizer like the one in GPT-4o-audio), but each decoder step has to attend over the full audio token stream, which is many times longer than the eventual transcript. The cost per minute of audio is meaningfully higher and the latency story is worse. On a bounded-input / bounded-output task like transcription, the encoder-decoder split wins on both axes. Whisper-large-v3, NVIDIA NeMo Canary, and Distil-Whisper are all production examples in 2026. Machine translation is the second strong fit. NLLB-200, M2M-100, and mBART use a shared encoder across many language pairs; bidirectional context on the source side helps disambiguate before the decoder commits. Decoder-only LLMs are very good at translation now too, but at high request volume for fixed-format pairs the encoder-decoder is still cheaper per token. **The structural property that drives all of this:** encoder-decoder wins when the input modality differs from the output, the input is bounded, the output is bounded, and the same input might drive multiple generations (re-decoding with different sampling temperatures, beam search, n-best candidates). The encoder pays once; the decoder reuses.

TL;DR

Speech-to-text. Whisper-style encoder-decoders run the encoder once on the audio, then the decoder cross-attends to that cached encoding at every step: far cheaper than a decoder-only LLM ingesting audio tokens

Memory aid

Easy to grasp

Imagine you have a long voice message and you want it written down. An encoder-decoder model is like having one specialist who listens to the whole recording from start to finish and writes a summary on a whiteboard, then a writer who looks at that whiteboard while typing the transcript word by word. The whiteboard is written once and read many times. A decoder-only LLM would be like asking the writer to re-read the entire audio from scratch before each word they type: wasteful when the audio never changes. That's the core reason Whisper and other speech models still beat decoder-only LLMs on cost and latency for transcription, even though decoder-only LLMs in 2026 are technically capable of doing it.

Key concepts

Attention

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

The transformer architecture in 2017 came in two flavors: the encoder-decoder (the original 'Attention Is All You Need' shape used for translation) and the decoder-only (GPT, used for left to right generation). For half a decade the field has consolidated around decoder-only LLMs, to the point where new practitioners often assume encoder-decoder is a legacy shape. It is not.

The correct framing for 2026 is that encoder-decoder owns a specific corner of the workload space: tasks where the input is bounded and reused across many decoder steps, where the input modality differs from the output modality, and where unit economics at scale matter more than open-ended generation quality. Speech-to-text is the textbook example, machine translation is the strong second, and bounded structured extraction is the quiet third.

This question is testing whether the candidate has internalized why the split exists in the first place, not just which models use which shape today.

What the encoder-decoder split actually does

An encoder-decoder transformer processes input in two distinct phases. The encoder stack runs once over the input sequence and produces a sequence of hidden states with shape [encoder_seq_len, d_model]. These hidden states encode bidirectional context (every position can attend to every other position during encoding) and they are then frozen for the duration of the query.

The decoder stack generates output autoregressively, one token at a time. Each decoder layer has two attention sub-blocks. Self-attention lets the decoder attend over the tokens it has already generated (masked to be causal). Cross-attention lets the decoder attend over the encoder's hidden states; the encoder hidden states are the K and V, the decoder's own hidden state at the current step is the Q.

The key property: the encoder runs once, the decoder runs many times. The encoder hidden states are computed once at the start of decoding and reused at every subsequent decoder step at zero recompute cost. For a 30-second audio clip that produces a 100-token transcript, the encoder runs once and the decoder runs 100 times against the same cached encoder hidden states.

A decoder-only transformer has no such split. The input and the output sit in one sequence, separated by special tokens. Every decoder step self-attends over the entire prefix, which includes the original input. The KV cache of the input is reused (so the input is not re-encoded), but the per step attention compute over that cached prefix is still paid every step.

Speech-to-text: the canonical 2026 fit

Translation: the strong second fit

The general structural rule

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Property	Encoder-decoder	Decoder-only LLM
Input encoding cost	Once per query	Once per query (but in same sequence)
Per-decode-step attention	Self-attn over text + cross-attn over cached encoder K/V	Self-attn over full audio + text prefix
Cost-per-audio-second	Low (Whisper-class)	Multiples higher (audio-LLM)
Best fit	Bounded input + bounded output (transcription, translation, extraction)	Open-ended generation, reasoning, multi-turn chat

Real products, models, and research that use this idea.

Whisper-large-v3 is the workhorse open-weight speech-to-text encoder-decoder in 2026; it powers podcast transcription pipelines, call-center analytics, and meeting-summary products.
Distil-Whisper is a distilled encoder-decoder variant that runs at roughly 6x the speed of Whisper-large for production transcription at scale.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is the audio-token sequence so much longer than the text-token sequence for the same content?

Speech tokenizers operate at frame rates of 25-50 Hz on the audio side, while text tokenizers compress to roughly 3-4 characters per token. A 30-second clip of speech becomes ~1500 audio tokens and ~75 text tokens. The 20x ratio is what makes encoder-decoder attractive for transcription.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Claiming decoder-only models have fully replaced encoder-decoders in 2026. They have not. Speech to text, large-batch translation, and bounded-output structured extraction still favor encoder-decoder on cost per token and latency.

60 second bullets to scan on the way to the call.

Cross-attention in the decoder and how it reuses the encoder's cached hidden states
Why audio token sequences are much longer than the resulting text sequences

Primary sources. Browse if you want the original framing.