Translating one source sentence into many target languages, what attention-side trick scales?

Attempt it

An encoder-decoder translation service receives N requests that all translate the same English source sentence into different target languages (French, Spanish, German, Japanese). Identify the attention-side optimization the runtime can apply, and explain why it is structurally available in encoder-decoder but only partially in decoder-only.

Free · 2 AI evals / day

Quick reference

TL;DR

Run the encoder once and reuse the cached cross-attention K, V across all N decoder generations; decoder-only can do this partially via prefix caching.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine a museum tour guide who has memorized a long script about one painting. If ten visitors all want to hear about that painting in ten different languages, the guide does not need to look at the painting ten times. They look once, form one mental description, and then translate that description into each language. The painting is the source sentence; the mental description is the encoder output; the translations are the decoder runs. Encoder-decoder transformers do exactly this. Decoder-only models work more like a guide who has to re-look at the painting every time they start a new translation, unless they have a clever notebook that remembers what they saw last time when the visitor description matches exactly.

Key concepts

Encoder-decoder architecture Cross-attention KV cache Prefix caching Decoder-only architecture

Quick scan · 2–4 min

Everything important, quickly.

Name the cross-attention K, V cache; explain why it is reusable across the N target generations; quantify the saving; connect prefix caching as the decoder-only analog; cite vLLM and SGLang as production implementations.

Side-by-side

Aspect	Encoder-decoder K, V reuse	Decoder-only prefix caching
Granularity of sharing	Whole encoder output	Exact token-level prefix match
Robustness to per-request diff	High, decoder state is independent	Low, any divergence ends sharing
Cache structure	Per-layer cross-attn K, V	Per-layer self-attn K, V blocks
Typical use case	Translation, ASR beam search	Chat with shared system prompt
Saving for N parallel runs	Encoder cost / N (large)	Depends on prefix overlap

Real-world examples

Real products, models, and research that use this idea.

Google's production NMT service for translation: encoder K, V cache shared across multi-target translation requests.
Whisper (encoder-decoder ASR): the audio encoder runs once per utterance and the decoder beam search reads the same cached cross-attention K, V for every hypothesis.
vLLM and SGLang in 2026 production: prefix caching for decoder-only models with system-prompt reuse; the closest analog to encoder K, V caching.
Hugging Face's `EncoderDecoderModel.generate` with `past_key_values`: caches cross-attention K, V across beam-search and multi-output generation.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is cross-attention K, V cacheable but cross-attention Q is not?

Sign in or go Pro to see the approach for this follow-up.

QWhat changes if the source is very long but each target is very short? Does the reuse still pay off?

Sign in or go Pro to see the approach for this follow-up.

QHow could you bring full encoder-K-V-style reuse to a decoder-only architecture?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.