Each row of the post-softmax attention weight matrix corresponds to which side of the QK product?

The attention score matrix is computed as Q times K transpose, where Q has shape (seq_len, d_k) and K has shape (seq_len, d_k). The result has shape (seq_len, seq_len). Standard matrix-multiplication convention: the rows of the output correspond to the rows of the LEFT operand (Q), and the columns correspond to the rows of the RIGHT operand (which is K, since we transposed it). So entry (i, j) in the score matrix is q_i dot k_j, the alignment between query i and key j. When we apply softmax along the LAST dimension (i.e., across columns of each row independently), each row becomes a probability distribution. The row indexed by i now contains the attention weights that query token i assigns to every key token j across the sequence. Each row sums to 1 because softmax normalizes each row. The semantic reading: row i tells you 'how much attention does the token at position i pay to every other position in the sequence?' Different queries pay attention differently, but every query splits 100 percent of its attention across the keys.

**Rows are queries, columns are keys.** The attention weight matrix has shape (seq_len_q, seq_len_k), and row i lists how much the query at position i attends to every key position j. Softmax is applied along the last dimension (the key dimension), so each row independently sums to exactly 1. The convention comes straight from the matrix arithmetic: $$\text{scores} = \frac{Q K^\top}{\sqrt{d_k}}$$ Q has shape (seq_len_q, d_k), K transposed has shape (d_k, seq_len_k), so the product has shape (seq_len_q, seq_len_k). Entry (i, j) is q_i dot k_j, the alignment score between query i and key j. After softmax over the row axis, entry (i, j) becomes a_ij, the fraction of query i's attention budget going to key j. The output is then `output_i = sum_j a_ij V_j`. The row i of the attention matrix is the recipe for combining the value vectors into the output for query position i. **Why this matters for masking.** Causal masks block positions j > i from contributing to row i. The mask is added to the score matrix before softmax; positions to be blocked get -inf so softmax sends them to 0. The row still sums to 1 because softmax renormalizes over the unmasked positions only.