Which config token names the count of parallel attention heads in a layer?

Attempt it

n_heads. In Hugging Face configs the canonical key is num_attention_heads (sometimes shortened to num_heads); papers usually write h or n_heads. All three names refer to the same number, how many independent (Q, K, V) projection slices run side by side inside one attention layer. It is distinct from num_kv_heads, which under GQA or MQA can be smaller than n_heads (each query head shares K, V with others in its group). A Llama-2 70B layer, for example, has 64 query heads but only 8 KV heads.

Your answer

The config token that names the count of parallel attention heads in a layer is **n_heads**. In Hugging Face config.json files the canonical key is `num_attention_heads`, sometimes shortened to `num_heads`. Papers usually write `h` or `n_heads`. All three names refer to the same number. What it represents: how many independent (Q, K, V) projection slices run side by side inside one attention layer. A model with `n_heads = 32` runs 32 parallel attention computations per layer, concatenates the per-head outputs, and passes the result through the output projection W_O. It is distinct from `num_kv_heads` (HF key `num_key_value_heads`), which appeared once GQA and MQA entered production. Under standard multi-head attention (MHA), `num_kv_heads = n_heads`. Under GQA, `num_kv_heads` is smaller, because groups of query heads share K and V. Under MQA, `num_kv_heads = 1`. A concrete example: Llama-2 70B has `num_attention_heads = 64` and `num_key_value_heads = 8`. There are 64 query heads but only 8 KV heads; each KV head is shared by 8 query heads. The KV cache size scales with `num_kv_heads`, not `n_heads`, which is why the distinction matters for production memory math.

The config token is **n_heads** (HuggingFace: `num_attention_heads`, sometimes `num_heads`; papers: `h`). It names the number of parallel attention heads in one attention layer. Each head runs an independent scaled dot-product attention computation in a `d_head = d_model / n_heads` subspace. The h per-head context vectors are concatenated back to width d_model and passed through the output projection `W_O`. Under standard MHA, each head has its own Q, K, and V projections, so `num_kv_heads` (HF: `num_key_value_heads`) equals `n_heads`. Under GQA, `num_kv_heads` is smaller, query heads are partitioned into groups and each group shares one K and one V. Under MQA, `num_kv_heads = 1`. Why the distinction matters: - Parameter count depends on `d_model`, not `n_heads`. Splitting d_model across more heads is a reshape, not new parameters. - KV cache size depends on `num_kv_heads`, not `n_heads`. The cache scales as `2 * L * num_kv_heads * d_head * T * b`. - Production decoders use GQA to keep `n_heads` high (for query-side specialization) and `num_kv_heads` low (for cache memory). Llama 4 Maverick keeps 64 query heads and 8 KV heads. The takeaway: knowing the config key `num_attention_heads` is the first step. Knowing how it relates to `num_key_value_heads`, `d_model`, `d_head`, and KV cache size is what makes the knowledge useful.

Quick reference

TL;DR

n_heads (or num_attention_heads in HuggingFace configs, h in papers) names the number of parallel (Q, K, V) projection slices in one attention layer.

Memory aid

Easy to grasp

Imagine the model's config file as a recipe card. One line says how many separate burners the attention kitchen runs in parallel. That number is n_heads, the count of independent attention paths through a single layer. Modern recipes also have a second number for how many shared sauce pots feed all those burners (num_kv_heads under GQA), but the burner count itself is n_heads.

Key concepts

Multi-head attention Grouped-query attention

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

3m: the canonical names (n_heads, num_attention_heads, h), what they control vs do not control, the split into num_kv_heads under GQA, and concrete production examples like Llama 4 Maverick's 64 query heads with 8 KV heads.

Side-by-side

Token	Where it appears	What it counts
n_heads	Papers, casual usage	Parallel attention heads per layer
num_attention_heads	HuggingFace config.json	Same as n_heads
num_heads	Some libraries (PyTorch nn.MultiheadAttention)	Same as n_heads
h	Math notation in papers	Same as n_heads
num_key_value_heads	HuggingFace config.json (GQA/MQA models)	Number of distinct K, V heads in the cache

Real-world examples

Real products, models, and research that use this idea.

Llama 4 Maverick's config has `num_attention_heads: 64` and `num_key_value_heads: 8`.
Llama-2 7B has `num_attention_heads: 32` and `num_key_value_heads: 32` (it is MHA, not GQA).
Mistral 7B has `num_attention_heads: 32` and `num_key_value_heads: 8` (GQA with group size 4).
DeepSeek V4 keeps a normal n_heads on the query side and uses MLA to compress K, V into a latent that occupies a single logical KV slot per layer.
Any HuggingFace transformers config file (`config.json`) exposes both keys for any GQA or MQA-equipped model.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QHow is `d_head` related to `n_heads` and `d_model`?

QWhy do modern Hugging Face configs expose num_key_value_heads as a separate key?

QWhat is the typical relationship between n_heads, d_head, and d_model in 2026 frontier models?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Confusing n_heads (query side) with num_kv_heads (cache side). Under MHA they are equal; under GQA or MQA they differ and conflating them gives wrong cache math.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.