Translating one source sentence into many target languages, what attention side trick scales?

Short answer·Medium·4.0 · 0·~3 min·Asked atDatabricks Tcs·Relevant atAi4bharat Cerebras Microsoft Replicate

Attempt it

An encoder-decoder translation service receives N requests that all translate the same English source sentence into different target languages (French, Spanish, German, Japanese). Identify the attention side optimization the runtime can apply, and explain why it is structurally available in encoder-decoder but only partially in decoder-only.

Free · 2 AI evals / day

The trick is **encoder output caching**. In an encoder-decoder transformer, the encoder produces one fixed representation of the source sentence. The decoder then reads from that representation via cross-attention. For a service translating one English sentence into N target languages, the runtime can: 1. Run the encoder ONCE on the English source. 2. Cache the per-layer cross-attention K and V derived from the encoder's output. 3. For each of the N target languages, run a separate decoder pass that reads the SAME cached K, V via cross-attention. This saves the encoder compute N - 1 times. For a Transformer-Base style model where the encoder is roughly half the work, that's a 30-50% latency cut per request beyond the first. Why decoder-only can only partially do this: decoder-only models do not have a separate encoder. The source sentence sits at the front of the same causal sequence the decoder generates into. Modern serving stacks (vLLM, SGLang) implement **prefix caching** that reuses the KV of the leading tokens across requests, but it requires identical leading tokens. Once the requests diverge (different target prefixes, different sampled tokens), the sharing ends.

Run the encoder **once** on the shared source and cache the per-layer cross-attention K and V derived from the encoder's final hidden states. All N decoder runs then read from the same cached K, V via cross-attention, skipping the encoder entirely. The encoder cost is paid once and amortized across all N generations. This is structurally clean in encoder-decoder because the encoder is a separate sub-network whose output depends only on the source. The decoder is a separate sub-network whose state is independent per request. The cache is safely shareable: nothing the decoder does can change what the encoder produced. Decoder-only architectures can only do this partially. There is no separate encoder; the source sentence lives as the prefix of the same causal sequence the decoder generates into. Its K, V live in the decoder's self-attention cache. **Prefix caching** in vLLM, SGLang, and TensorRT-LLM reuses those prefix K, V across requests that share leading tokens. But the constraint is exact-match: two requests with even one different leading token share nothing from that point on. For the translation use case, encoder-decoder gives near-perfect reuse regardless of target language; decoder-only requires the user prompt to be byte-identical across all N requests, which is brittle in practice.

TL;DR

Run the encoder once and reuse the cached cross-attention K, V across all N decoder generations; decoder-only can do this partially via prefix caching.

Memory aid

Easy to grasp

Imagine a museum tour guide who has memorized a long script about one painting. If ten visitors all want to hear about that painting in ten different languages, the guide does not need to look at the painting ten times. They look once, form one mental description, and then translate that description into each language. The painting is the source sentence; the mental description is the encoder output; the translations are the decoder runs. Encoder-decoder transformers do exactly this. Decoder-only models work more like a guide who has to re-look at the painting every time they start a new translation, unless they have a clever notebook that remembers what they saw last time when the visitor description matches exactly.

Key concepts

Encoder-decoder architecture Cross-attention KV cache Prefix caching Decoder-only architecture

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Encoder K, V caching is the cleanest example of architectural amortization in transformer serving. It only works in encoder-decoder models, it only matters when one source feeds multiple downstream generations, and when those conditions hit it can cut serving cost by 30-50%. Decoder-only architectures dominate the LLM market because their workloads are usually one-shot per source, but the moment you have heavy source reuse, translation, ASR beam search, multi-modal cross-attention into a fixed image, the encoder-decoder design's caching story becomes structurally superior.

This deep dive walks the mechanics of cross-attention K, V caching, quantifies the savings, contrasts with prefix caching in modern decoder-only serving stacks, and closes with a guide for when the encoder-decoder design is structurally the right choice.

What is actually cached

When people say 'cache the encoder output' for translation, they are being imprecise. Let's get specific.

The encoder produces hidden states

Given a source sentence of L tokens, the encoder produces a (L, d_model) tensor at each of its layers. The final-layer output is the one the decoder consumes.

The decoder's cross-attention layers project K and V

In each decoder block, the cross-attention sub-layer has its own W_K and W_V matrices. These project the encoder's final-layer hidden states to produce (L, d_model) K and V tensors per decoder layer.

Those K and V are what gets cached

For a decoder with D layers, the cache is D pairs of (L, num_heads, d_head) tensors, one (K, V) per cross-attention sub-layer per layer.

Why this granularity matters

The Q in cross-attention is projected from the decoder hidden state, which changes per generated token. So Q must be re-computed every step. But K and V depend only on the encoder output, which is fixed once the source is set. So they can be projected once and reused for every Q across every decoder step across every target language.

The cache is per-layer per-(K or V). Not per-head, not per-token, the granularity is layer-shaped tensors.

Quantifying the win

Why decoder-only cannot do this exactly

Prefix caching in modern serving stacks

Decision guide: which architecture for which workload

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Aspect	Encoder-decoder K, V reuse	Decoder-only prefix caching
Granularity of sharing	Whole encoder output	Exact token-level prefix match
Robustness to per-request diff	High, decoder state is independent	Low, any divergence ends sharing
Cache structure	Per-layer cross-attn K, V	Per-layer self-attn K, V blocks
Typical use case	Translation, ASR beam search	Chat with shared system prompt
Saving for N parallel runs	Encoder cost / N (large)	Depends on prefix overlap

Real products, models, and research that use this idea.

Google's production NMT service for translation: encoder K, V cache shared across multi-target translation requests.
Whisper (encoder-decoder ASR): the audio encoder runs once per utterance and the decoder beam search reads the same cached cross-attention K, V for every hypothesis.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy is cross-attention K, V cacheable but cross-attention Q is not?

K and V depend only on the encoder output, which depends only on the source. Q is projected from the decoder hidden state, which changes for every generated token in every target language. So Q must be re-computed per request and per token; K and V are fixed once the source is set.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Claiming decoder-only can do exact encoder-output reuse via prefix caching. Prefix caching only works for exact token-level prefix matches; cross-attention reuse works regardless of how decoder runs diverge.

60 second bullets to scan on the way to the call.

What gets cached in encoder-decoder reuse (per-layer cross-attention K, V)
Why the encoder pass is reusable (output depends only on source)

Primary sources. Browse if you want the original framing.