Tying K and V to one projection, what is gained and what is risked?

MCQ·Medium·4.0 · 0·~1 min

Attempt it

Standard attention computes three projections from each input vector x: Q = xW_Q, K = xW_K, V = xW_V. Q is the query (what this token is looking for), K is the key (what each token advertises so it can be found), V is the value (what content gets returned when a token is selected). Tying K and V means using one projection: W_K = W_V, so K and V are the same tensor. You compute it once and use it for both the dot-product matching and the weighted sum. The gains are mechanical. The attention block has one fewer projection matrix, cutting that part of the parameter count roughly in half. More importantly for inference, the KV cache stores only one tensor per layer, head, position, halving the cache footprint and the memory bandwidth consumed during decode. The risk is structural. In standard attention K and V play different roles. K is optimized to be findable by relevant queries (matching). V is optimized to carry useful content to the next layer (delivery). Tying them forces one vector to do both jobs, and the model loses the ability to learn 'find me by attribute X, then return content Y'. Quality drops measurably, which is why frontier models almost never tie K and V. MQA and GQA give you most of the memory win without the quality cost by sharing K and V across heads instead of merging them.

**Tied K and V** is a structural shortcut where W_K = W_V, so the key tensor and the value tensor are literally identical. You compute one projection per token instead of two. The gains are concrete. Attention parameter count drops by the size of one projection matrix, typically a few percent of total model parameters. The KV cache stores one tensor instead of two per (layer, head, position), halving the per-token cache footprint. For a 70B-class model that's the difference between ~327 KB per token and ~164 KB per token at FP16. The structural cost is also concrete. The canonical retrieval framing of attention is: Q is the query (what am I looking for?), K is the key/address (what makes me findable?), V is the value/payload (what do I deliver when found?). The model uses Q dot K to decide weights and then takes a weighted sum of V. Tying K = V means the address and the payload are the same vector. The model loses the ability to learn mappings like 'a token is findable by syntactic role X but delivers semantic content Y'. In practice this hurts. Empirically, models with tied K = V underperform standard attention on language modeling by a few percent perplexity, which translates to meaningful quality drops on downstream tasks. **Why frontier models don't do this.** They use MQA (one KV head shared across all query heads) or GQA (a few KV heads grouped across query heads) instead. These share K and V ACROSS heads but keep K and V as distinct tensors per group. You get most of the memory win without merging the address and payload roles.

Quick reference

TL;DR

Tying K and V halves attention parameters and KV-cache memory, but forces lookup address and content payload to be the same vector, which measurably hurts quality.

Memory aid

Easy to grasp

Imagine a library where each book's title doubles as its entire contents. To find any book you search by title, which is fine, but every time you find one, the only thing inside is its own title repeated. The library saves a lot of shelf space, but the books are useless because the lookup label and the actual material people want to read are forced to be identical. Real libraries (and real transformers) keep titles separate from book contents so each can be optimized for its own job, finding things versus delivering useful material.

Key concepts

QKV projections KV cache Multi-query attention Grouped-query attention

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5m: parameter and memory accounting for tied K = V, why K and V have different gradient signals, structural cost vs head-level sharing, comparison table MHA vs tied vs MQA vs GQA vs MLA, and why no frontier model uses tied K = V.

Side-by-side

Technique	What's shared	KV-cache savings	Quality cost
Standard MHA	Nothing	Baseline	Baseline
Tied K = V	K and V are the same tensor per head	2x	1 to 3% perplexity loss
MQA	One KV head for all query heads	n_heads-x	Measurable but small
GQA	KV heads grouped across query heads (typically 8 groups)	4x to 8x	Negligible at scale
MLA	K and V compressed into shared latent space	10x or more	Negligible after kernel tuning

Real-world examples

Real products, models, and research that use this idea.

Most production transformers (GPT-style decoders, BERT-style encoders, T5 encoder-decoder) keep K and V as distinct projections; tied K = V is rare in published work.
Some early on-device transformer designs (MobileBERT variants, some tiny LLMs in 2022-2023) used tied K = V to fit aggressive memory budgets, accepting quality loss.
Llama 4 Maverick, Claude Opus 4.7, Gemini 3.1 Pro all use GQA, keeping K and V distinct but sharing KV heads across query heads.
DeepSeek V4 uses Multi-head Latent Attention (MLA), compressing K and V into a low-rank latent space, an alternative to tying them entirely.
Mistral Large 3 uses sliding-window attention combined with GQA; nowhere in the design is K tied to V.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhy does GQA work well but tied K = V doesn't, given both are forms of weight sharing in attention?

QCould you partially tie K and V, e.g., share a few dimensions while keeping others separate?

QHow would you measure the quality cost of tied K = V in your own model?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Confusing tied K = V with GQA or MQA. MQA and GQA share K and V across heads while keeping K and V as distinct tensors; tying K = V makes them literally the same vector, a much more aggressive cut.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.