Which config token names the count of parallel attention heads in a layer?

Flashcard·Easy·4.0 · 0·~30s·Asked atLepton Ai Pinterest Salesforce·Relevant atMicrosoft

Attempt it

n_heads. In Hugging Face configs the canonical key is num_attention_heads (sometimes shortened to num_heads); papers usually write h or n_heads. All three names refer to the same number, how many independent (Q, K, V) projection slices run side by side inside one attention layer. It is distinct from num_kv_heads, which under GQA or MQA can be smaller than n_heads (each query head shares K, V with others in its group). A Llama-2 70B layer, for example, has 64 query heads but only 8 KV heads.

Your answer

The config token that names the count of parallel attention heads in a layer is **n_heads**. In Hugging Face config.json files the canonical key is `num_attention_heads`, sometimes shortened to `num_heads`. Papers usually write `h` or `n_heads`. All three names refer to the same number. What it represents: how many independent (Q, K, V) projection slices run side by side inside one attention layer. A model with `n_heads = 32` runs 32 parallel attention computations per layer, concatenates the per-head outputs, and passes the result through the output projection W_O. It is distinct from `num_kv_heads` (HF key `num_key_value_heads`), which appeared once GQA and MQA entered production. Under standard multi-head attention (MHA), `num_kv_heads = n_heads`. Under GQA, `num_kv_heads` is smaller, because groups of query heads share K and V. Under MQA, `num_kv_heads = 1`. A concrete example: Llama-2 70B has `num_attention_heads = 64` and `num_key_value_heads = 8`. There are 64 query heads but only 8 KV heads; each KV head is shared by 8 query heads. The KV cache size scales with `num_kv_heads`, not `n_heads`, which is why the distinction matters for production memory math.

The config token is **n_heads** (HuggingFace: `num_attention_heads`, sometimes `num_heads`; papers: `h`). It names the number of parallel attention heads in one attention layer. Each head runs an independent scaled dot-product attention computation in a `d_head = d_model / n_heads` subspace. The h per-head context vectors are concatenated back to width d_model and passed through the output projection `W_O`. Under standard MHA, each head has its own Q, K, and V projections, so `num_kv_heads` (HF: `num_key_value_heads`) equals `n_heads`. Under GQA, `num_kv_heads` is smaller, query heads are partitioned into groups and each group shares one K and one V. Under MQA, `num_kv_heads = 1`. Why the distinction matters: - Parameter count depends on `d_model`, not `n_heads`. Splitting d_model across more heads is a reshape, not new parameters. - KV cache size depends on `num_kv_heads`, not `n_heads`. The cache scales as `2 * L * num_kv_heads * d_head * T * b`. - Production decoders use GQA to keep `n_heads` high (for query-side specialization) and `num_kv_heads` low (for cache memory). Llama 4 Maverick keeps 64 query heads and 8 KV heads. The takeaway: knowing the config key `num_attention_heads` is the first step. Knowing how it relates to `num_key_value_heads`, `d_model`, `d_head`, and KV cache size is what makes the knowledge useful.

TL;DR

n_heads (or num_attention_heads in HuggingFace configs, h in papers) names the number of parallel (Q, K, V) projection slices in one attention layer.

Memory aid

Easy to grasp

Imagine the model's config file as a recipe card. One line says how many separate burners the attention kitchen runs in parallel. That number is n_heads, the count of independent attention paths through a single layer. Modern recipes also have a second number for how many shared sauce pots feed all those burners (num_kv_heads under GQA), but the burner count itself is n_heads.

Key concepts

Multi-head attention Grouped-query attention

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

The config token that names the count of parallel attention heads is one of the first numbers anyone learning transformer architecture meets. It looks simple: a single integer per attention layer. But the naming conventions vary across papers and frameworks, and the relationship between this token and the parameter count, the KV cache size, and the GQA-era separate num_key_value_heads is one of the most common sources of confusion in interview and design conversations.

The canonical name is n_heads, with HuggingFace exposing it as num_attention_heads and papers using h. All three refer to the same thing: how many parallel (Q, K, V) projection slices run side by side in one attention layer.

This deep dive walks the naming conventions across paper, code, and config; what n_heads actually controls (and what it does not); the modern split into n_heads vs num_kv_heads under GQA and MQA; and the concrete 2026 production examples that make the distinction concrete.

The three names for the same number

The number of parallel attention heads appears in slightly different forms across the ecosystem.

In papers

h is the standard math notation. The Vaswani 2017 paper uses h throughout. Most subsequent papers follow.
n_heads appears in many ablation tables and prose.
Sometimes spelled num_heads in figures and pseudocode.

In HuggingFace configs (`config.json`)

num_attention_heads: the canonical key for the query side.
Some older configs use num_heads.
GQA and MQA models also have num_key_value_heads, which is a separate count.

In PyTorch and other libraries

torch.nn.MultiheadAttention uses num_heads.
Megatron and DeepSpeed use num_attention_heads.
vLLM and TensorRT-LLM read from the HuggingFace config and propagate the same keys.

What they all mean

In every case, the number names how many parallel attention paths run through one layer. The Q, K, V projections of width d_model are reshaped into (n_heads, d_head) blocks, each block runs its own scaled dot-product attention, and the outputs are concatenated and passed through W_O.

Knowing all three names lets you read papers, configs, and code interchangeably without getting tripped up by terminology.

What n_heads controls (and what it does NOT)

The split into n_heads vs num_kv_heads under GQA

Why this matters for parameter and cache math

Why d_head usually stays around 64 or 128

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Token	Where it appears	What it counts
n_heads	Papers, casual usage	Parallel attention heads per layer
num_attention_heads	HuggingFace config.json	Same as n_heads
num_heads	Some libraries (PyTorch nn.MultiheadAttention)	Same as n_heads
h	Math notation in papers	Same as n_heads
num_key_value_heads	HuggingFace config.json (GQA/MQA models)	Number of distinct K, V heads in the cache

Real products, models, and research that use this idea.

Llama 4 Maverick's config has `num_attention_heads: 64` and `num_key_value_heads: 8`.
Llama-2 7B has `num_attention_heads: 32` and `num_key_value_heads: 32` (it is MHA, not GQA).

What an interviewer would ask next. Try answering before peeking at the approach.

QHow is `d_head` related to `n_heads` and `d_model`?

d_head = d_model / n_heads. The split is exact; production models pick d_model and n_heads such that d_head is a clean integer (often 64 or 128). Wider d_head per head usually correlates with smaller n_heads at the same d_model.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Confusing n_heads (query side) with num_kv_heads (cache side). Under MHA they are equal; under GQA or MQA they differ and conflating them gives wrong cache math.

60 second bullets to scan on the way to the call.

The three names for the same concept: n_heads, num_attention_heads, h
What n_heads controls: parallel (Q, K, V) projection slices per layer

Primary sources. Browse if you want the original framing.