Standard attention computes three projections from each input vector x: Q = xW_Q, K = xW_K, V = xW_V. Q is the query (what this token is looking for), K is the key (what each token advertises so it can be found), V is the value (what content gets returned when a token is selected).
Tying K and V means using one projection: W_K = W_V, so K and V are the same tensor. You compute it once and use it for both the dot-product matching and the weighted sum.
The gains are mechanical. The attention block has one fewer projection matrix, cutting that part of the parameter count roughly in half. More importantly for inference, the KV cache stores only one tensor per layer, head, position, halving the cache footprint and the memory bandwidth consumed during decode.
The risk is structural. In standard attention K and V play different roles. K is optimized to be findable by relevant queries (matching). V is optimized to carry useful content to the next layer (delivery). Tying them forces one vector to do both jobs, and the model loses the ability to learn 'find me by attribute X, then return content Y'. Quality drops measurably, which is why frontier models almost never tie K and V. MQA and GQA give you most of the memory win without the quality cost by sharing K and V across heads instead of merging them.
**Tied K and V** is a structural shortcut where W_K = W_V, so the key tensor and the value tensor are literally identical. You compute one projection per token instead of two.
The gains are concrete. Attention parameter count drops by the size of one projection matrix, typically a few percent of total model parameters. The KV cache stores one tensor instead of two per (layer, head, position), halving the per-token cache footprint. For a 70B-class model that's the difference between ~327 KB per token and ~164 KB per token at FP16.
The structural cost is also concrete. The canonical retrieval framing of attention is: Q is the query (what am I looking for?), K is the key/address (what makes me findable?), V is the value/payload (what do I deliver when found?). The model uses Q dot K to decide weights and then takes a weighted sum of V. Tying K = V means the address and the payload are the same vector. The model loses the ability to learn mappings like 'a token is findable by syntactic role X but delivers semantic content Y'.
In practice this hurts. Empirically, models with tied K = V underperform standard attention on language modeling by a few percent perplexity, which translates to meaningful quality drops on downstream tasks.
**Why frontier models don't do this.** They use MQA (one KV head shared across all query heads) or GQA (a few KV heads grouped across query heads) instead. These share K and V ACROSS heads but keep K and V as distinct tensors per group. You get most of the memory win without merging the address and payload roles.
Quick reference
TL;DR
Tying K and V halves attention parameters and KV-cache memory, but forces lookup address and content payload to be the same vector, which measurably hurts quality.
Imagine a library where each book's title doubles as its entire contents. To find any book you search by title, which is fine, but every time you find one, the only thing inside is its own title repeated. The library saves a lot of shelf space, but the books are useless because the lookup label and the actual material people want to read are forced to be identical. Real libraries (and real transformers) keep titles separate from book contents so each can be optimized for its own job, finding things versus delivering useful material.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Frontier-quality models targeting the best possible perplexity per parameter Tied K = V costs 1 to 3 percent perplexity vs separate projections, a regression that compounds across tasks; GQA or MLA give you most of the memory savings without the quality hit.
Models with long-context retrieval requirements (needle in haystack, RAG over long documents) The K vs V distinction is most useful when the model needs to find one specific token by an attribute different from the content it delivers; tying them removes that flexibility exactly where it matters most.
Production serving stacks that already use GQA or MLA These give you 4x to 8x cache reduction (GQA) or further compression (MLA) without the structural cost; layering tied K = V on top would compound quality losses for diminishing returns.
Encoder models for embeddings or classification, where attention patterns matter heavily Encoders rely on rich, specialized attention patterns; tying K = V collapses the expressivity that makes BERT-style and ViT-style embeddings work.
Quick scan · 2–4 min
Everything important, quickly.
5m: parameter and memory accounting for tied K = V, why K and V have different gradient signals, structural cost vs head-level sharing, comparison table MHA vs tied vs MQA vs GQA vs MLA, and why no frontier model uses tied K = V.
Side-by-side
Technique
What's shared
KV-cache savings
Quality cost
Standard MHA
Nothing
Baseline
Baseline
Tied K = V
K and V are the same tensor per head
2x
1 to 3% perplexity loss
MQA
One KV head for all query heads
n_heads-x
Measurable but small
GQA
KV heads grouped across query heads (typically 8 groups)
4x to 8x
Negligible at scale
MLA
K and V compressed into shared latent space
10x or more
Negligible after kernel tuning
Real-world examples
Real products, models, and research that use this idea.
1Most production transformers (GPT-style decoders, BERT-style encoders, T5 encoder-decoder) keep K and V as distinct projections; tied K = V is rare in published work.
2Some early on-device transformer designs (MobileBERT variants, some tiny LLMs in 2022-2023) used tied K = V to fit aggressive memory budgets, accepting quality loss.
3Llama 4 Maverick, Claude Opus 4.7, Gemini 3.1 Pro all use GQA, keeping K and V distinct but sharing KV heads across query heads.
4DeepSeek V4 uses Multi-head Latent Attention (MLA), compressing K and V into a low-rank latent space, an alternative to tying them entirely.
5Mistral Large 3 uses sliding-window attention combined with GQA; nowhere in the design is K tied to V.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does GQA work well but tied K = V doesn't, given both are forms of weight sharing in attention?
Sign in or go Pro to see the approach for this follow-up.
QCould you partially tie K and V, e.g., share a few dimensions while keeping others separate?
Sign in or go Pro to see the approach for this follow-up.
QHow would you measure the quality cost of tied K = V in your own model?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Confusing tied K = V with GQA or MQA. MQA and GQA share K and V across heads while keeping K and V as distinct tensors; tying K = V makes them literally the same vector, a much more aggressive cut.