In the post-softmax attention weight matrix, each row corresponds to one token and contains that token's attention weights over all tokens. Every row sums to .
The attention score matrix is computed as Q times K transpose, where Q has shape (seq_len, d_k) and K has shape (seq_len, d_k). The result has shape (seq_len, seq_len). Standard matrix-multiplication convention: the rows of the output correspond to the rows of the LEFT operand (Q), and the columns correspond to the rows of the RIGHT operand (which is K, since we transposed it).
So entry (i, j) in the score matrix is q_i dot k_j, the alignment between query i and key j.
When we apply softmax along the LAST dimension (i.e., across columns of each row independently), each row becomes a probability distribution. The row indexed by i now contains the attention weights that query token i assigns to every key token j across the sequence. Each row sums to 1 because softmax normalizes each row.
The semantic reading: row i tells you 'how much attention does the token at position i pay to every other position in the sequence?' Different queries pay attention differently, but every query splits 100 percent of its attention across the keys.
**Rows are queries, columns are keys.** The attention weight matrix has shape (seq_len_q, seq_len_k), and row i lists how much the query at position i attends to every key position j. Softmax is applied along the last dimension (the key dimension), so each row independently sums to exactly 1.
The convention comes straight from the matrix arithmetic:
$$\text{scores} = \frac{Q K^\top}{\sqrt{d_k}}$$
Q has shape (seq_len_q, d_k), K transposed has shape (d_k, seq_len_k), so the product has shape (seq_len_q, seq_len_k). Entry (i, j) is q_i dot k_j, the alignment score between query i and key j. After softmax over the row axis, entry (i, j) becomes a_ij, the fraction of query i's attention budget going to key j.
The output is then `output_i = sum_j a_ij V_j`. The row i of the attention matrix is the recipe for combining the value vectors into the output for query position i.
**Why this matters for masking.** Causal masks block positions j > i from contributing to row i. The mask is added to the score matrix before softmax; positions to be blocked get -inf so softmax sends them to 0. The row still sums to 1 because softmax renormalizes over the unmasked positions only.
Quick reference
TL;DR
Each row corresponds to one query token; entries along the row are that token's weights over all key tokens; row sums to 1 because softmax normalizes per row.
Picture a table where each row belongs to one person who is asking a question. That row lists how much attention this person pays to every other person in the room. The numbers in the row are slices of a pie that has to add up to a whole pie, because the person is dividing 100 percent of their attention across the room. Different rows belong to different askers, each splitting their own pie independently.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Sparse attention variants where rows must sum over a custom-defined sparse set The row-sum-to-1 invariant still holds, but only over the unmasked columns; debugging a sparse pattern requires confirming the mask is correct before trusting the row sum.
Multi-query and grouped-query attention with multiple Q heads per KV head The row-axis convention is unchanged but the shape gets confusing: each Q head still produces (T_q, T_k) rows, but multiple Q heads share the same K and V, so debugging requires tracking head groupings explicitly.
Cross-attention where T_q (decoder tokens) differs from T_k (encoder tokens) The attention matrix is rectangular; some debugging tools assume square. Each decoder query still gets one row summing to 1 over encoder positions.
FlashAttention or other fused kernels where the full attention matrix is never materialized You cannot directly inspect the per-row weights; the kernel computes the output stream by stream and the matrix exists only conceptually. Use a non-fused implementation for inspection.
Quick scan · 2–4 min
Everything important, quickly.
4m: matrix shape from QK^T, softmax along the key axis, row-sum-to-1 invariant, mask interaction, decode-time degenerate case, and operational consequences for visualization and FlashAttention.
Real-world examples
Real products, models, and research that use this idea.
1Attention visualization tools like BertViz and the HuggingFace inspect APIs all use the row=query, column=key convention.
2FlashAttention 2 and 3 tile over rows of Q in the outer loop, processing one query block at a time against streamed K blocks.
3Llama 4 Maverick's GQA still has the same row-sum-to-1 invariant per query head, even though multiple query heads share KV heads.
4Sliding-window attention in Mistral and similar models defines a mask that keeps only nearby columns per row; row sums still equal 1 over the window.
5StreamingLLM with attention sinks keeps the first few key columns plus a sliding window; rows sum to 1 over sink + window positions.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhat changes about the matrix during autoregressive decoding when T_q equals 1?
Sign in or go Pro to see the approach for this follow-up.
QHow does the row convention interact with multi-head attention?
Sign in or go Pro to see the approach for this follow-up.
QWhy does the row-sum invariant hold even with causal masking?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Swapping rows and columns. Rows index queries (who is asking) and columns index keys (who is being asked about). Flipping this breaks every subsequent operation.