Where does cross-attention live in OpenAI Whisper?

MCQ·Medium·4.0 · 0·~1 min·Relevant atOpenAI

Attempt it

Whisper is an encoder-decoder transformer designed for speech recognition. It has two main components: an audio encoder and a text decoder. The audio encoder takes raw audio, converts it to a mel-spectrogram (a 2D representation of frequencies over time), and runs self-attention over the mel frames to produce a sequence of latent audio embeddings. This is pure self-attention within the audio modality; no cross-attention here. The text decoder generates the transcript token by token, autoregressively. Inside each decoder block, there are THREE attention-related operations: - **Masked self-attention** over the text tokens generated so far (causal, so each token only sees past tokens). - **Cross-attention**, where queries come from the text stream and keys/values come from the encoder's audio embeddings. This is the bridge between modalities. - A feed-forward network, like any transformer block. The cross-attention layer is what lets each text token 'look at' the audio to decide what to write next. Without it, the decoder would be generating text from nothing; with it, the decoder is grounded in the actual audio content. Whisper is NOT decoder-only. The original Whisper paper from OpenAI in 2022 explicitly uses an encoder-decoder design because the audio input (continuous, dense) and the text output (discrete tokens) have very different characteristics that benefit from separate encoding and decoding pipelines.

**Whisper is an encoder-decoder transformer.** The audio encoder processes mel-spectrogram frames with self-attention to produce a sequence of audio embeddings. The text decoder generates the transcript autoregressively, and cross-attention is what bridges the two modalities. Inside each decoder block, the order of operations is: 1. **Masked self-attention** over the text tokens generated so far. The mask is causal; each token attends only to past tokens. 2. **Cross-attention** with queries from the current decoder hidden states and keys/values from the encoder's output. This is where the decoder pulls relevant audio information into each text token's representation. 3. **Feed-forward network**, identical to a standard transformer block. The shapes make the bridge concrete. Suppose the audio is 30 seconds, encoded into 1500 mel frames at 50 Hz. After the encoder, you have a (1500, d_model) tensor of audio embeddings. The decoder's text stream might be 50 tokens long; queries from those 50 tokens dot-product against the 1500 audio keys, producing a (50, 1500) attention matrix per head. The weighted sum over the 1500 audio values gives each text token a context-rich representation grounded in the relevant audio frames. **The wrong answers reflect common misconceptions:** - The encoder's self-attention isn't cross-attention; both Q and K come from the same audio stream. - Whisper doesn't fuse audio into token embeddings before attention; the fusion happens inside the decoder via cross-attention. - Whisper is explicitly encoder-decoder. Some newer speech models (Speech-LLaMA, audio-LLM hybrids) are decoder-only with prepended audio tokens, but Whisper isn't one of them.

Quick reference

TL;DR

Cross-attention lives inside each decoder block: queries come from the text token stream, keys and values come from the audio encoder's output.

Memory aid

Easy to grasp

Picture a court stenographer who listens to a recording and types up what was said. Their fingers (the decoder) are deciding the next character to type, but every few keystrokes they replay a bit of the audio to remember exactly what came next. The 'replaying audio while typing' is cross-attention: the typing side asks the audio side 'what should I write now?', and the audio side hands back the relevant sound, not as raw audio, but as a structured summary the typing side knows how to read.

Key concepts

Cross-attention Encoder-decoder architecture Whisper ASR model Speech recognition

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5m: Whisper's encoder-decoder architecture, where cross-attention sits in each decoder block, Q vs K/V sourcing, why encoder K/V are cached and reused, comparison to decoder-only audio LLMs, and the 30-second chunking trade-offs.

Code example

python

import torch
import torch.nn as nn

class WhisperDecoderBlock(nn.Module):
    def __init__(self, d_model, n_heads):
        super().__init__()
        self.self_attn = nn.MultiheadAttention(d_model, n_heads)
        self.cross_attn = nn.MultiheadAttention(d_model, n_heads)
        self.ffn = nn.Sequential(
            nn.Linear(d_model, 4 * d_model),
            nn.GELU(),
            nn.Linear(4 * d_model, d_model),
        )
        self.ln1 = nn.LayerNorm(d_model)
        self.ln2 = nn.LayerNorm(d_model)
        self.ln3 = nn.LayerNorm(d_model)

    def forward(self, x, audio_enc, causal_mask):
        # x: (T_text, B, d_model), decoder hidden state
        # audio_enc: (T_audio, B, d_model), encoder output (fixed per audio)

        # 1. Masked self-attention over text tokens
        h, _ = self.self_attn(x, x, x, attn_mask=causal_mask)
        x = self.ln1(x + h)

        # 2. Cross-attention: Q from text, K and V from audio encoder
        h, _ = self.cross_attn(query=x, key=audio_enc, value=audio_enc)
        x = self.ln2(x + h)

        # 3. Feed-forward
        x = self.ln3(x + self.ffn(x))
        return x

Real-world examples

Real products, models, and research that use this idea.

OpenAI Whisper large-v3 (released late 2023) is the production-default for high-quality ASR in 2026, used in transcription startups, podcast indexing, and meeting recorders.
WhisperX adds forced-alignment to Whisper's output for precise word-level timestamps, useful for subtitle generation.
Hugging Face Distil-Whisper is a distilled version that runs faster while preserving most of the quality, often deployed for on-device ASR.
T5 uses the same encoder-decoder + cross-attention pattern but for text to text tasks instead of speech to text.
DETR (object detection) and BART (summarization) also use this encoder-decoder + cross-attention design, showing the pattern transfers across modalities.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy are the encoder's K and V cached and reused across decode steps in Whisper?

QHow does Whisper handle audio longer than 30 seconds?

QWhat would change if Whisper were redesigned as a decoder-only model?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Confusing cross-attention with the encoder's self-attention. Self-attention happens within one modality (audio attending to audio); cross-attention bridges two modalities (text queries pulling from audio keys/values).

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.