Tying K and V to one projection, what is gained and what is risked?

MCQ·Medium·4.0 · 0·~1 min·Asked atCerebras Sap Turing·Relevant atMeta

Attempt it

Standard attention computes three projections from each input vector x: Q = xW_Q, K = xW_K, V = xW_V. Q is the query (what this token is looking for), K is the key (what each token advertises so it can be found), V is the value (what content gets returned when a token is selected). Tying K and V means using one projection: W_K = W_V, so K and V are the same tensor. You compute it once and use it for both the dot-product matching and the weighted sum. The gains are mechanical. The attention block has one fewer projection matrix, cutting that part of the parameter count roughly in half. More importantly for inference, the KV cache stores only one tensor per layer, head, position, halving the cache footprint and the memory bandwidth consumed during decode. The risk is structural. In standard attention K and V play different roles. K is optimized to be findable by relevant queries (matching). V is optimized to carry useful content to the next layer (delivery). Tying them forces one vector to do both jobs, and the model loses the ability to learn 'find me by attribute X, then return content Y'. Quality drops measurably, which is why frontier models almost never tie K and V. MQA and GQA give you most of the memory win without the quality cost by sharing K and V across heads instead of merging them.

**Tied K and V** is a structural shortcut where W_K = W_V, so the key tensor and the value tensor are literally identical. You compute one projection per token instead of two. The gains are concrete. Attention parameter count drops by the size of one projection matrix, typically a few percent of total model parameters. The KV cache stores one tensor instead of two per (layer, head, position), halving the per-token cache footprint. For a 70B-class model that's the difference between ~327 KB per token and ~164 KB per token at FP16. The structural cost is also concrete. The canonical retrieval framing of attention is: Q is the query (what am I looking for?), K is the key/address (what makes me findable?), V is the value/payload (what do I deliver when found?). The model uses Q dot K to decide weights and then takes a weighted sum of V. Tying K = V means the address and the payload are the same vector. The model loses the ability to learn mappings like 'a token is findable by syntactic role X but delivers semantic content Y'. In practice this hurts. Empirically, models with tied K = V underperform standard attention on language modeling by a few percent perplexity, which translates to meaningful quality drops on downstream tasks. **Why frontier models don't do this.** They use MQA (one KV head shared across all query heads) or GQA (a few KV heads grouped across query heads) instead. These share K and V ACROSS heads but keep K and V as distinct tensors per group. You get most of the memory win without merging the address and payload roles.

TL;DR

Tying K and V halves attention parameters and KV-cache memory, but forces lookup address and content payload to be the same vector, which measurably hurts quality.

Memory aid

Easy to grasp

Imagine a library where each book's title doubles as its entire contents. To find any book you search by title, which is fine, but every time you find one, the only thing inside is its own title repeated. The library saves a lot of shelf space, but the books are useless because the lookup label and the actual material people want to read are forced to be identical. Real libraries (and real transformers) keep titles separate from book contents so each can be optimized for its own job, finding things versus delivering useful material.

Key concepts

QKV projections KV cache Multi-query attention Grouped-query attention

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Tying K and V is the most aggressive form of weight sharing inside an attention block: instead of two separate projections producing two separate tensors, you compute one projection and use the result for both the key role (matching against queries) and the value role (delivering content to the weighted sum). The mechanical savings are immediate, half the projection parameters and half the KV cache, and the structural cost is also immediate: the model loses the ability to separate 'how to find me' from 'what to deliver when found'.

The reason this question is interesting is that it sits at a fork in the design tree. One branch (tied K = V) is rarely taken; the other branches (MQA, GQA, MLA) are the production defaults in 2026. Understanding why everyone walks the second branch reveals the right way to think about weight sharing in attention.

This deep dive walks through the mechanical accounting, the gradient signals that pull K and V apart during training, the empirical quality cost of forcing them together, and the alternatives that share intelligently along axes with redundancy rather than along axes with structural distinction.

Mechanical accounting: what tied K = V actually saves

Standard multi-head attention has four projection matrices per layer: W_Q, W_K, W_V, W_O. Each has shape (d_model, d_head * n_heads). For a Llama-style model with d_model = 4096, d_head = 128, n_heads = 32, each matrix has about 16.8M parameters; across a 32-layer model that's 538M per matrix type.

Parameter savings

Tying W_K = W_V eliminates one full matrix. For the example above, that's 538M parameters saved, about 7 percent of a 7B-class model. The relative savings shrink as the model grows because attention is a smaller fraction of total parameters in larger models (FFN dominates).

KV cache savings

The inference-time savings are usually more important than the parameter savings. The KV cache stores K and V for every (layer, head, position):

\text{bytes}_{\text{cache}} = 2 \cdot L \cdot H_{kv} \cdot d_h \cdot T \cdot b

Tying K = V means you only store one tensor per slot, dropping the leading factor of 2 to 1. For a 70B-class model with GQA at 8 KV heads, d_head = 128, 80 layers, FP16, the per-token footprint drops from about 327 KB to about 164 KB.

What's NOT saved

The QK^T matmul and the AV matmul both still happen at full size: the cost is in matrix multiplications over tensors of full shape, not in the projections. So tied K = V saves the projection FLOPs but doesn't change attention's O(seq_len^2 * d_head) core cost.

Why K and V want to be different vectors

Empirical quality cost

The right way to share: MQA, GQA, MLA

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Technique	What's shared	KV-cache savings	Quality cost
Standard MHA	Nothing	Baseline	Baseline
Tied K = V	K and V are the same tensor per head	2x	1 to 3% perplexity loss
MQA	One KV head for all query heads	n_heads-x	Measurable but small
GQA	KV heads grouped across query heads (typically 8 groups)	4x to 8x	Negligible at scale
MLA	K and V compressed into shared latent space	10x or more	Negligible after kernel tuning

Real products, models, and research that use this idea.

Most production transformers (GPT-style decoders, BERT-style encoders, T5 encoder-decoder) keep K and V as distinct projections; tied K = V is rare in published work.
Some early on-device transformer designs (MobileBERT variants, some tiny LLMs in 2022-2023) used tied K = V to fit aggressive memory budgets, accepting quality loss.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does GQA work well but tied K = V doesn't, given both are forms of weight sharing in attention?

The two share at different axes. GQA shares across heads, where the model has demonstrable redundancy (many heads learn similar patterns). Tied K = V shares within a head between the address role and the payload role, where the model genuinely needs distinct specialization. Sharing along an axis with redundancy is cheap; sharing along an axis with structural distinction is expensive.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Confusing tied K = V with GQA or MQA. MQA and GQA share K and V across heads while keeping K and V as distinct tensors; tying K = V makes them literally the same vector, a much more aggressive cut.

60 second bullets to scan on the way to the call.

What K does in the attention computation versus what V does
Exact parameter and memory savings of tying W_K = W_V

Primary sources. Browse if you want the original framing.