Attention temperature divides QK^T; sampling temperature divides output logits, distinguishwhere each lives.

Attempt it

Quick reference

TL;DR

Attention temperature shapes mid-layer routing over keys; sampling temperature shapes final next-token selection over vocabulary. Different softmaxes, fully independent.

Memory aid

Sign in for the mnemonic that makes this stick. See Pro pricing.

Easy to grasp

Imagine writing a story. First, you keep notes on every paragraph you have already written so you can refer back. The way you decide which old paragraph to look at when writing a new sentence is one knob, that is attention temperature. Set it low and you laser-focus on one prior paragraph; set it high and you blend many. Second, when you actually pick the next word to write, you have a long list of possible words and you sample one. That sampling knob is sampling temperature. Set it low and you pick the most likely word; set it high and you pick something unusual. Two completely separate decisions, two completely separate knobs.

Key concepts

Softmax Attention temperature Sampling temperature Scaled dot-product attention

Quick scan · 2–4 min

Everything important, quickly.

6-8 min: pipeline softmax count + T_attn location and effect + T_sample location and effect + sqrt(d_k) vs T_attn distinction + production API exposure + failure mode comparison.

Side-by-side

Property	Attention temperature	Sampling temperature
Where it acts	Pre-softmax scores in every attention layer	Final vocabulary logits before sampling
What it shapes	Key routing per head	Next-token discrete choice
Failure mode (high)	Uniform attention, routing collapse	Random vocabulary, nonsense output
Failure mode (low)	Single-key lock, hallucination	Greedy repetition
API exposure	Rarely exposed	Always exposed

Real-world examples

Real products, models, and research that use this idea.

OpenAI, Anthropic, and Google LLM APIs (GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro) all expose sampling temperature as a user-facing knob; attention temperature is never exposed.
Research stacks like nanoGPT and academic transformer implementations sometimes expose attention temperature for ablation studies.
Llama 4 Maverick, DeepSeek V4, and Qwen 3.5 default sampling temperature is typically 0.7 to 1.0 for chat workloads.
Top-p, top-k, and min-p sampling combine with T_sample at the output; none of those interact with T_attn.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat is the relationship between attention temperature and the sqrt(d_k) factor?

Sign in or go Pro to see the approach for this follow-up.

QIf you set T_sample to zero, what happens?

Sign in or go Pro to see the approach for this follow-up.

QHow do top-p and top-k interact with sampling temperature?

Sign in or go Pro to see the approach for this follow-up.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

Sign in for 5 unlocks/day or go Pro for unlimited revision sheets. See Pro pricing.

References

Primary sources. Skim if you want the original framing.