Translating one source sentence into many target languages, what attention-side trick scales?

Short answer·Medium·4.0 · 0·~3 min

Attempt it

An encoder-decoder translation service receives N requests that all translate the same English source sentence into different target languages (French, Spanish, German, Japanese). Identify the attention-side optimization the runtime can apply, and explain why it is structurally available in encoder-decoder but only partially in decoder-only.

Free · 2 AI evals / day

The trick is **encoder output caching**. In an encoder-decoder transformer, the encoder produces one fixed representation of the source sentence. The decoder then reads from that representation via cross-attention. For a service translating one English sentence into N target languages, the runtime can: 1. Run the encoder ONCE on the English source. 2. Cache the per-layer cross-attention K and V derived from the encoder's output. 3. For each of the N target languages, run a separate decoder pass that reads the SAME cached K, V via cross-attention. This saves the encoder compute N - 1 times. For a Transformer-Base style model where the encoder is roughly half the work, that's a 30-50% latency cut per request beyond the first. Why decoder-only can only partially do this: decoder-only models do not have a separate encoder. The source sentence sits at the front of the same causal sequence the decoder generates into. Modern serving stacks (vLLM, SGLang) implement **prefix caching** that reuses the KV of the leading tokens across requests, but it requires identical leading tokens. Once the requests diverge (different target prefixes, different sampled tokens), the sharing ends.

Run the encoder **once** on the shared source and cache the per-layer cross-attention K and V derived from the encoder's final hidden states. All N decoder runs then read from the same cached K, V via cross-attention, skipping the encoder entirely. The encoder cost is paid once and amortized across all N generations. This is structurally clean in encoder-decoder because the encoder is a separate sub-network whose output depends only on the source. The decoder is a separate sub-network whose state is independent per request. The cache is safely shareable: nothing the decoder does can change what the encoder produced. Decoder-only architectures can only do this partially. There is no separate encoder; the source sentence lives as the prefix of the same causal sequence the decoder generates into. Its K, V live in the decoder's self-attention cache. **Prefix caching** in vLLM, SGLang, and TensorRT-LLM reuses those prefix K, V across requests that share leading tokens. But the constraint is exact-match: two requests with even one different leading token share nothing from that point on. For the translation use case, encoder-decoder gives near-perfect reuse regardless of target language; decoder-only requires the user prompt to be byte-identical across all N requests, which is brittle in practice.

Quick reference

TL;DR

Run the encoder once and reuse the cached cross-attention K, V across all N decoder generations; decoder-only can do this partially via prefix caching.

Memory aid

Easy to grasp

Imagine a museum tour guide who has memorized a long script about one painting. If ten visitors all want to hear about that painting in ten different languages, the guide does not need to look at the painting ten times. They look once, form one mental description, and then translate that description into each language. The painting is the source sentence; the mental description is the encoder output; the translations are the decoder runs. Encoder-decoder transformers do exactly this. Decoder-only models work more like a guide who has to re-look at the painting every time they start a new translation, unless they have a clever notebook that remembers what they saw last time when the visitor description matches exactly.

Key concepts

Encoder-decoder architecture Cross-attention KV cache Prefix caching Decoder-only architecture

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

Name the cross-attention K, V cache; explain why it is reusable across the N target generations; quantify the saving; connect prefix caching as the decoder-only analog; cite vLLM and SGLang as production implementations.

Side-by-side

Aspect	Encoder-decoder K, V reuse	Decoder-only prefix caching
Granularity of sharing	Whole encoder output	Exact token-level prefix match
Robustness to per-request diff	High, decoder state is independent	Low, any divergence ends sharing
Cache structure	Per-layer cross-attn K, V	Per-layer self-attn K, V blocks
Typical use case	Translation, ASR beam search	Chat with shared system prompt
Saving for N parallel runs	Encoder cost / N (large)	Depends on prefix overlap

Real-world examples

Real products, models, and research that use this idea.

Google's production NMT service for translation: encoder K, V cache shared across multi-target translation requests.
Whisper (encoder-decoder ASR): the audio encoder runs once per utterance and the decoder beam search reads the same cached cross-attention K, V for every hypothesis.
vLLM and SGLang in 2026 production: prefix caching for decoder-only models with system-prompt reuse; the closest analog to encoder K, V caching.
Hugging Face's `EncoderDecoderModel.generate` with `past_key_values`: caches cross-attention K, V across beam-search and multi-output generation.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is cross-attention K, V cacheable but cross-attention Q is not?

QWhat changes if the source is very long but each target is very short? Does the reuse still pay off?

QHow could you bring full encoder-K-V-style reuse to a decoder-only architecture?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Claiming decoder-only can do exact encoder-output reuse via prefix caching. Prefix caching only works for exact token-level prefix matches; cross-attention reuse works regardless of how decoder runs diverge.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.