Name a 2026 task where an encoder-decoder transformer still beats a decoder-only LLM

Short answer·Easy·4.0 · 0·~3 min·Relevant atGoogle Mistral AI NVIDIA

Attempt it

Pick one production task where an encoder-decoder transformer is still the right choice in 2026, and explain what specifically makes encoder-decoder a better fit than a decoder-only LLM.

Free · 2 AI evals / day

Encoder-decoder transformers split the work into two stacks. The **encoder** reads the input once and builds a representation. The **decoder** generates the output one token at a time while cross-attending to that fixed encoder representation. The canonical 2026 fit is speech-to-text. Whisper, NVIDIA NeMo Canary, and Distil-Whisper all use this shape. The encoder ingests a mel-spectrogram of the audio in one pass and produces a sequence of hidden states. The decoder then emits text tokens one by one, cross-attending to that cached encoder output at every step. The audio is never re-encoded; the same representation drives the entire transcript. A decoder-only LLM in 2026 (GPT-5.5, Llama 3.1 70B, Claude) can technically transcribe audio when paired with an audio tokenizer, but each decoder step has to attend over the full audio token stream, which is much longer than the corresponding text. The cost per second of audio is higher and the latency is worse on a fixed-format task like transcription where the input is bounded and the output is bounded. Translation is the second classic fit. NLLB-200 and M2M-100 serve dozens of language pairs over a single shared encoder. Bidirectional context on the source side helps disambiguate words before the decoder commits to a target word.

**Speech-to-text** is the canonical 2026 fit for encoder-decoder. The encoder ingests the mel-spectrogram once and produces a sequence of hidden states; the decoder autoregressively emits text while cross-attending to that cached encoding. The audio representation is built once and reused at every decoder step at zero recompute cost. A decoder-only LLM can technically do transcription in 2026 (with an audio tokenizer like the one in GPT-4o-audio), but each decoder step has to attend over the full audio token stream, which is many times longer than the eventual transcript. The cost per minute of audio is meaningfully higher and the latency story is worse. On a bounded-input / bounded-output task like transcription, the encoder-decoder split wins on both axes. Whisper-large-v3, NVIDIA NeMo Canary, and Distil-Whisper are all production examples in 2026. Machine translation is the second strong fit. NLLB-200, M2M-100, and mBART use a shared encoder across many language pairs; bidirectional context on the source side helps disambiguate before the decoder commits. Decoder-only LLMs are very good at translation now too, but at high request volume for fixed-format pairs the encoder-decoder is still cheaper per token. **The structural property that drives all of this:** encoder-decoder wins when the input modality differs from the output, the input is bounded, the output is bounded, and the same input might drive multiple generations (re-decoding with different sampling temperatures, beam search, n-best candidates). The encoder pays once; the decoder reuses.

Quick reference

TL;DR

Speech-to-text. Whisper-style encoder-decoders run the encoder once on the audio, then the decoder cross-attends to that cached encoding at every step: far cheaper than a decoder-only LLM ingesting audio tokens

Memory aid

Easy to grasp

Imagine you have a long voice message and you want it written down. An encoder-decoder model is like having one specialist who listens to the whole recording from start to finish and writes a summary on a whiteboard, then a writer who looks at that whiteboard while typing the transcript word by word. The whiteboard is written once and read many times. A decoder-only LLM would be like asking the writer to re-read the entire audio from scratch before each word they type: wasteful when the audio never changes. That's the core reason Whisper and other speech models still beat decoder-only LLMs on cost and latency for transcription, even though decoder-only LLMs in 2026 are technically capable of doing it.

Key concepts

Attention

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: encoder-decoder architecture and cross-attention, the cost per audio second argument, Whisper / NeMo Canary / Distil-Whisper as concrete 2026 examples, why NLLB-class translation still beats decoder-only on bulk serving, and the bounded-input / bounded-output property that drives the win.

Side-by-side

Property	Encoder-decoder	Decoder-only LLM
Input encoding cost	Once per query	Once per query (but in same sequence)
Per-decode-step attention	Self-attn over text + cross-attn over cached encoder K/V	Self-attn over full audio + text prefix
Cost-per-audio-second	Low (Whisper-class)	Multiples higher (audio-LLM)
Best fit	Bounded input + bounded output (transcription, translation, extraction)	Open-ended generation, reasoning, multi-turn chat

Real-world examples

Real products, models, and research that use this idea.

Whisper-large-v3 is the workhorse open-weight speech-to-text encoder-decoder in 2026; it powers podcast transcription pipelines, call-center analytics, and meeting-summary products.
Distil-Whisper is a distilled encoder-decoder variant that runs at roughly 6x the speed of Whisper-large for production transcription at scale.
NVIDIA NeMo Canary is the 2024-2026 encoder-decoder model topping the Hugging Face Open ASR Leaderboard for English transcription.
Meta's NLLB-200 serves 200 language-pair translation from a shared encoder; subtitle and localization pipelines at scale use it because it is cheaper per token than a decoder-only LLM.
T5 and BART variants still ship in 2026 for structured extraction tasks: invoice parsing, biomedical entity linking, schema-constrained JSON generation.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is the audio-token sequence so much longer than the text-token sequence for the same content?

QCan a decoder-only LLM cache audio tokens the way encoder-decoder caches encoder hidden states?

QWhy has translation moved partially toward decoder-only LLMs despite encoder-decoder's structural fit?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Claiming decoder-only models have fully replaced encoder-decoders in 2026. They have not. Speech to text, large-batch translation, and bounded-output structured extraction still favor encoder-decoder on cost per token and latency.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.