Which config token names the count of parallel attention heads in a layer?

The config token that names the count of parallel attention heads in a layer is **n_heads**. In Hugging Face config.json files the canonical key is `num_attention_heads`, sometimes shortened to `num_heads`. Papers usually write `h` or `n_heads`. All three names refer to the same number. What it represents: how many independent (Q, K, V) projection slices run side by side inside one attention layer. A model with `n_heads = 32` runs 32 parallel attention computations per layer, concatenates the per-head outputs, and passes the result through the output projection W_O. It is distinct from `num_kv_heads` (HF key `num_key_value_heads`), which appeared once GQA and MQA entered production. Under standard multi-head attention (MHA), `num_kv_heads = n_heads`. Under GQA, `num_kv_heads` is smaller, because groups of query heads share K and V. Under MQA, `num_kv_heads = 1`. A concrete example: Llama-2 70B has `num_attention_heads = 64` and `num_key_value_heads = 8`. There are 64 query heads but only 8 KV heads; each KV head is shared by 8 query heads. The KV cache size scales with `num_kv_heads`, not `n_heads`, which is why the distinction matters for production memory math.

The config token is **n_heads** (HuggingFace: `num_attention_heads`, sometimes `num_heads`; papers: `h`). It names the number of parallel attention heads in one attention layer. Each head runs an independent scaled dot-product attention computation in a `d_head = d_model / n_heads` subspace. The h per-head context vectors are concatenated back to width d_model and passed through the output projection `W_O`. Under standard MHA, each head has its own Q, K, and V projections, so `num_kv_heads` (HF: `num_key_value_heads`) equals `n_heads`. Under GQA, `num_kv_heads` is smaller, query heads are partitioned into groups and each group shares one K and one V. Under MQA, `num_kv_heads = 1`. Why the distinction matters: - Parameter count depends on `d_model`, not `n_heads`. Splitting d_model across more heads is a reshape, not new parameters. - KV cache size depends on `num_kv_heads`, not `n_heads`. The cache scales as `2 * L * num_kv_heads * d_head * T * b`. - Production decoders use GQA to keep `n_heads` high (for query-side specialization) and `num_kv_heads` low (for cache memory). Llama 4 Maverick keeps 64 query heads and 8 KV heads. The takeaway: knowing the config key `num_attention_heads` is the first step. Knowing how it relates to `num_key_value_heads`, `d_model`, `d_head`, and KV cache size is what makes the knowledge useful.