Where does cross-attention live in OpenAI Whisper?

Attempt it

Quick reference

TL;DR

Cross-attention lives inside each decoder block: queries come from the text token stream, keys and values come from the audio encoder's output.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Picture a court stenographer who listens to a recording and types up what was said. Their fingers (the decoder) are deciding the next character to type, but every few keystrokes they replay a bit of the audio to remember exactly what came next. The 'replaying audio while typing' is cross-attention: the typing side asks the audio side 'what should I write now?', and the audio side hands back the relevant sound, not as raw audio, but as a structured summary the typing side knows how to read.

Key concepts

Cross-attention Encoder-decoder architecture Whisper ASR model Speech recognition

Quick scan · 2–4 min

Everything important, quickly.

5m: Whisper's encoder-decoder architecture, where cross-attention sits in each decoder block, Q vs K/V sourcing, why encoder K/V are cached and reused, comparison to decoder-only audio LLMs, and the 30-second chunking trade-offs.

Code example

python

import torch
import torch.nn as nn

class WhisperDecoderBlock(nn.Module):
    def __init__(self, d_model, n_heads):
        super().__init__()
        self.self_attn = nn.MultiheadAttention(d_model, n_heads)
        self.cross_attn = nn.MultiheadAttention(d_model, n_heads)
        self.ffn = nn.Sequential(
            nn.Linear(d_model, 4 * d_model),
            nn.GELU(),
            nn.Linear(4 * d_model, d_model),
        )
        self.ln1 = nn.LayerNorm(d_model)
        self.ln2 = nn.LayerNorm(d_model)
        self.ln3 = nn.LayerNorm(d_model)

    def forward(self, x, audio_enc, causal_mask):
        # x: (T_text, B, d_model), decoder hidden state
        # audio_enc: (T_audio, B, d_model), encoder output (fixed per audio)

        # 1. Masked self-attention over text tokens
        h, _ = self.self_attn(x, x, x, attn_mask=causal_mask)
        x = self.ln1(x + h)

        # 2. Cross-attention: Q from text, K and V from audio encoder
        h, _ = self.cross_attn(query=x, key=audio_enc, value=audio_enc)
        x = self.ln2(x + h)

        # 3. Feed-forward
        x = self.ln3(x + self.ffn(x))
        return x

Real-world examples

Real products, models, and research that use this idea.

OpenAI Whisper large-v3 (released late 2023) is the production-default for high-quality ASR in 2026, used in transcription startups, podcast indexing, and meeting recorders.
WhisperX adds forced-alignment to Whisper's output for precise word-level timestamps, useful for subtitle generation.
Hugging Face Distil-Whisper is a distilled version that runs faster while preserving most of the quality, often deployed for on-device ASR.
T5 uses the same encoder-decoder + cross-attention pattern but for text to text tasks instead of speech to text.
DETR (object detection) and BART (summarization) also use this encoder-decoder + cross-attention design, showing the pattern transfers across modalities.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy are the encoder's K and V cached and reused across decode steps in Whisper?

Sign in or go Pro to see the approach for this follow-up.

QHow does Whisper handle audio longer than 30 seconds?

Sign in or go Pro to see the approach for this follow-up.

QWhat would change if Whisper were redesigned as a decoder-only model?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.