Walk through how speech or sound becomes something a language backbone can attend to. Cover the front end representation, the encoder, and how the result reaches the LLM, and contrast that with a natively speech to speech model.
Waveform becomes a log-mel spectrogram, a Whisper-style encoder turns it into embeddings, a connector maps those into LLM token space — mirroring the vision pipeline. Native audio models skip the text transcript.
Imagine turning a song into a heat map: time runs left to right, pitch runs bottom to top, and bright spots show which notes are loud when. That heat map is something a computer can study much more easily than a raw squiggle of sound. A reader (the audio encoder) skims that heat map and writes down a tidy summary the language model can understand. The slow way is to first transcribe everything to plain words, which throws away tone and emotion. The fast, modern way keeps the sound itself, so the model still hears whether you sounded angry or joking.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Voice features feel magical when they work, but under the hood the question is mundane: an LLM attends over a sequence of vectors, and sound is not a sequence of vectors. Something has to bridge the gap. The interesting part is that the bridge looks almost identical to the one used for images, and that the choice of whether to route through a text transcript is a real architectural fork with consequences users can hear.
Interviewers ask this to check two things: that you know the concrete front-end to LLM path, and that you understand why 2026 voice assistants moved away from transcribe then respond toward native audio.
This deep dive walks the four-stage pipeline, explains why the spectrogram front-end exists, shows how the connector parallels vision, and then digs into the cascaded versus native tradeoff that decides whether a voice product feels like a conversation or like dictation.
Why a spectrogram, not the raw waveform
A raw waveform is a long list of amplitude samples — 16,000 numbers per second is common. Feeding that directly to a transformer is wasteful: the sequence is enormous and the useful structure, which lives in frequency content over time, is buried in the raw oscillation.
The fix is a log-mel spectrogram. You slide a short window across the signal, take a Fourier transform in each window to get the frequency content, map those frequencies onto the mel scale that matches human pitch perception, and take the log of the magnitudes. The result is a 2D image: time on one axis, frequency bins on the other, brightness showing energy.
This representation is both compact and informative. It collapses tens of thousands of raw samples per second into a manageable grid where the patterns that distinguish phonemes, instruments, and speakers are visible. It is the audio equivalent of splitting an image into patches — turning messy raw signal into structured input an encoder can chew on.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- OpenAI Whisper — the canonical log-mel spectrogram plus transformer audio encoder.
- Native audio to audio voice — pioneered in 2024 by OpenAI's first omni model and carried forward by the 2026 flagship GPT-5.5 voice, which skips the transcript for low-latency speech.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow do native audio models actually represent and generate audio tokens?
Discuss discrete audio codecs or learned audio tokenizers that turn sound into a token stream the model can model and decode.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Stopping at speech to text. The cascaded transcript path drops prosody and adds latency; native audio-token models keep tone and skip the text bottleneck.
60 second bullets to scan on the way to the call.
Why raw waveform is converted to a log-mel spectrogram first
What a Whisper-style audio encoder outputs
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.