Temperature

Walk through how top_p (nucleus) sampling truncates the distribution

Compare top_k and top_p as truncation strategies for sampling

How does the temperature parameter reshape the sampling distribution?

What does dividing pre-softmax attention scores by an extra factor > 1 do at inference?

What happens to attention weights when pre-softmax scores are divided by a temperature T > 1?

If attention temperature divides QK^T and sampling temperature divides the output logits, where is each one applied?

What does the temperature parameter do during LLM generation and how should you set it?

When does temperature > 0 produce better results than temperature = 0 in production LLM applications?

Why does self-consistency require temperature > 0 and what specifically happens if you set T = 0?

Explain why self-consistency requires temperature > 0 and quantify the gain it produces over single pass CoT on math benchmarks.

Spot the error in this explanation of temperature.