Perplexity
Also known as: PPL
Exp(average cross-entropy) on held-out text; lower means the model is less surprised by real data.
An intrinsic evaluation metric for language models: the exponential of the average negative log-likelihood per token. Lower perplexity means the model assigns higher probability to held-out text: a better generative fit.
In practice
Standard metric for pre-training quality. Interviews probe its weakness (poor proxy for instruction-following) and why it's not used for chat eval.
How it compares
Perplexity is one specific automated metric; LLM evaluation is the umbrella practice spanning many.
Related topics
Questions that mention this term
- Decompose the cost of a single API call into its components and explain which dominates.
- What is the primary purpose of embeddings in modern NLP?
- Dividing pre-softmax attention scores by an extra factor > 1 at inference does what?
- Order the pipeline that stretches an 8k pretrained model to 128k context
- Find the flaw: lower pretraining perplexity guarantees better chat quality
- Why does pure long context FT (without RoPE scaling) fail to extrapolate?
Related terms
Chatbot Arena
Crowdsourced blind A/B comparisons of LLM responses, aggregated into an Elo leaderboard.
Eval Harness
A framework for running, scoring, and tracking LLM evals over time; handles golden sets, judges, and regressions.
Golden Set
Curated test cases with expected outputs; your regression suite for prompt and model changes.
Guardrails
Pre- and post-processing layers that block bad inputs/outputs and enforce policy on top of an LLM.
Hallucination
When a model confidently makes up something that isn't true.
HumanEval
Code-generation benchmark: 164 problems with hidden unit tests, scored by whether the generated code passes.