Where does cross-attention live in OpenAI Whisper?

MCQ·Medium·4.0 · 0·~1 min·Asked atDescript Figure Ai·Relevant atOpenAI

Attempt it

Whisper is an encoder-decoder transformer designed for speech recognition. It has two main components: an audio encoder and a text decoder. The audio encoder takes raw audio, converts it to a mel-spectrogram (a 2D representation of frequencies over time), and runs self-attention over the mel frames to produce a sequence of latent audio embeddings. This is pure self-attention within the audio modality; no cross-attention here. The text decoder generates the transcript token by token, autoregressively. Inside each decoder block, there are THREE attention-related operations: - **Masked self-attention** over the text tokens generated so far (causal, so each token only sees past tokens). - **Cross-attention**, where queries come from the text stream and keys/values come from the encoder's audio embeddings. This is the bridge between modalities. - A feed-forward network, like any transformer block. The cross-attention layer is what lets each text token 'look at' the audio to decide what to write next. Without it, the decoder would be generating text from nothing; with it, the decoder is grounded in the actual audio content. Whisper is NOT decoder-only. The original Whisper paper from OpenAI in 2022 explicitly uses an encoder-decoder design because the audio input (continuous, dense) and the text output (discrete tokens) have very different characteristics that benefit from separate encoding and decoding pipelines.

**Whisper is an encoder-decoder transformer.** The audio encoder processes mel-spectrogram frames with self-attention to produce a sequence of audio embeddings. The text decoder generates the transcript autoregressively, and cross-attention is what bridges the two modalities. Inside each decoder block, the order of operations is: 1. **Masked self-attention** over the text tokens generated so far. The mask is causal; each token attends only to past tokens. 2. **Cross-attention** with queries from the current decoder hidden states and keys/values from the encoder's output. This is where the decoder pulls relevant audio information into each text token's representation. 3. **Feed-forward network**, identical to a standard transformer block. The shapes make the bridge concrete. Suppose the audio is 30 seconds, encoded into 1500 mel frames at 50 Hz. After the encoder, you have a (1500, d_model) tensor of audio embeddings. The decoder's text stream might be 50 tokens long; queries from those 50 tokens dot-product against the 1500 audio keys, producing a (50, 1500) attention matrix per head. The weighted sum over the 1500 audio values gives each text token a context-rich representation grounded in the relevant audio frames. **The wrong answers reflect common misconceptions:** - The encoder's self-attention isn't cross-attention; both Q and K come from the same audio stream. - Whisper doesn't fuse audio into token embeddings before attention; the fusion happens inside the decoder via cross-attention. - Whisper is explicitly encoder-decoder. Some newer speech models (Speech-LLaMA, audio-LLM hybrids) are decoder-only with prepended audio tokens, but Whisper isn't one of them.

TL;DR

Cross-attention lives inside each decoder block: queries come from the text token stream, keys and values come from the audio encoder's output.

Memory aid

Easy to grasp

Picture a court stenographer who listens to a recording and types up what was said. Their fingers (the decoder) are deciding the next character to type, but every few keystrokes they replay a bit of the audio to remember exactly what came next. The 'replaying audio while typing' is cross-attention: the typing side asks the audio side 'what should I write now?', and the audio side hands back the relevant sound, not as raw audio, but as a structured summary the typing side knows how to read.

Key concepts

Cross-attention Encoder-decoder architecture Whisper ASR model Speech recognition

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Whisper is OpenAI's open-source automatic speech recognition (ASR) system, released in late 2022 and widely deployed in production through 2026 as the default high-quality ASR option. It uses the classic encoder-decoder transformer architecture from the original Vaswani 2017 paper, with one twist: the encoder processes audio (mel-spectrogram frames) instead of text. Cross-attention is what bridges the two modalities, lives in every decoder block, and is the single most important architectural detail for understanding how Whisper turns audio into text.

The question of 'where does cross-attention live in Whisper' is really a question about the encoder-decoder pattern itself. Cross-attention is the second attention block inside each decoder layer (between masked self-attention and the FFN), and it pulls queries from the decoder's text stream while pulling keys and values from the encoder's audio embeddings. That asymmetric Q vs K/V sourcing is what defines cross-attention and distinguishes it from self-attention.

This deep dive walks through Whisper's architecture, the mechanics of cross-attention inside the decoder, why encoder K/V are cached and reused, the 30-second chunking design, and the comparison to newer decoder-only audio LLMs.

Whisper's encoder-decoder architecture

Whisper has two main components: an audio encoder and a text decoder. They share architectural style (transformer blocks) but operate on different input modalities.

The audio encoder

Input: 30 seconds of audio converted to a log-mel spectrogram of shape (n_mels=80, n_frames=3000) at 100 Hz. A 2-layer 1D conv stem downsamples to 1500 frames at 50 Hz, projecting from 80 mel channels to d_model. Then learned positional embeddings are added, and 6 to 32 transformer blocks (depending on model size) process the audio with full bidirectional self-attention.

Output: a (1500, d_model) tensor of audio embeddings. This is what the decoder will attend over.

The text decoder

The decoder generates transcript tokens autoregressively. Each decoder block has three sub-blocks:

Masked self-attention over the text tokens generated so far. The mask is causal; each token attends only to past tokens.
Cross-attention with Q from the decoder hidden states and K, V from the encoder output. This is where the decoder pulls audio information.
Feed-forward network, identical to standard transformer FFN.

LayerNorm is applied before each sub-block (pre-norm variant). The decoder uses the same number of layers as the encoder.

The vocabulary

50k+ multilingual BPE tokens, plus special tokens for language identification (50 languages plus 'auto'), task selection (transcribe vs translate), and timestamp prediction. The special tokens are critical: they let one model handle multiple languages and both ASR and translation.

Cross-attention mechanics: Q from text, K and V from audio

Why encoder K and V are cached and reused

Encoder-decoder vs decoder-only: the design choice

Situations where this technique stops working.

2–4 min · Everything important, quickly.

python

import torch
import torch.nn as nn

class WhisperDecoderBlock(nn.Module):
    def __init__(self, d_model, n_heads):
        super().__init__()
        self.self_attn = nn.MultiheadAttention(d_model, n_heads)
        self.cross_attn = nn.MultiheadAttention(d_model, n_heads)
        self.ffn = nn.Sequential(
            nn.Linear(d_model, 4 * d_model),
            nn.GELU(),
            nn.Linear(4 * d_model, d_model),
        )
        self.ln1 = nn.LayerNorm(d_model)
        self.ln2 = nn.LayerNorm(d_model)
        self.ln3 = nn.LayerNorm(d_model)

    def forward(self, x, audio_enc, causal_mask):
        # x: (T_text, B, d_model), decoder hidden state
        # audio_enc: (T_audio, B, d_model), encoder output (fixed per audio)

        # 1. Masked self-attention over text tokens
        h, _ = self.self_attn(x, x, x, attn_mask=causal_mask)
        x = self.ln1(x + h)

        # 2. Cross-attention: Q from text, K and V from audio encoder
        h, _ = self.cross_attn(query=x, key=audio_enc, value=audio_enc)
        x = self.ln2(x + h)

        # 3. Feed-forward
        x = self.ln3(x + self.ffn(x))
        return x

Real products, models, and research that use this idea.

OpenAI Whisper large-v3 (released late 2023) is the production-default for high-quality ASR in 2026, used in transcription startups, podcast indexing, and meeting recorders.
WhisperX adds forced-alignment to Whisper's output for precise word-level timestamps, useful for subtitle generation.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy are the encoder's K and V cached and reused across decode steps in Whisper?

The encoder output is a fixed function of the audio input; it doesn't depend on the text being generated. Computing K and V from the encoder output once and reusing them for every decoder step avoids redundant work. This is the cross-attention analog of the KV cache in decoder-only models, but for the encoder side. It's why Whisper's per-token decode latency is low: the expensive part (encoding 30 seconds of audio with multiple transformer layers) is amortized across all transcript tokens.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Confusing cross-attention with the encoder's self-attention. Self-attention happens within one modality (audio attending to audio); cross-attention bridges two modalities (text queries pulling from audio keys/values).

60 second bullets to scan on the way to the call.

Whisper's architecture: encoder-decoder, not decoder-only
Audio encoder pipeline: mel-spectrogram, conv stem, self-attention layers

Primary sources. Browse if you want the original framing.