What does the dot product q . k measure inside an attention layer?

The dot product `q . k` is a single number that measures **how aligned** two vectors are. Geometrically: `q . k = |q| * |k| * cos(theta)`, where theta is the angle between q and k. Three cases that build intuition: 1. q and k point the same way → cos(theta) is near 1 → the dot product is large and positive. 2. q and k are perpendicular → cos(theta) is 0 → the dot product is 0. 3. q and k point opposite ways → cos(theta) is near -1 → the dot product is large and negative. The magnitudes matter too. Two short vectors that align perfectly produce a smaller dot product than two long vectors that align perfectly. This is one reason transformers divide by `sqrt(d_k)` afterward, to keep score magnitudes stable as the dimension grows. In an attention layer, q is the query (what this token wants) and k is the key (what some other token advertises). The dot product is the raw 'compatibility score'. After scaling and softmax, it becomes a weight between 0 and 1, telling the model how much of that key's value vector to mix into the output.

The raw dot product `q . k` is the unnormalized affinity between a query and a key. It captures two things at once: the angular alignment between the vectors (via cosine) and their magnitudes: $$q \cdot k = \lVert q \rVert \, \lVert k \rVert \, \cos(\theta)$$ When q and k point the same way the score is large and positive. When they are orthogonal it is zero. When they oppose it is negative. Magnitudes matter as much as direction. A perfectly aligned pair of unit vectors only scores 1, while two longer aligned vectors score larger. In attention, the dot product is the affinity signal, nothing more. Two transforms follow: - **Scale by `1 / sqrt(d_k)`** to stop variance from blowing up as the head dimension grows. Without scaling, softmax saturates and gradients vanish at high `d_k`. - **Apply softmax** across all keys to turn raw scores into a probability distribution over which keys to read from. So the dot product is the 'how well does this query match this key' answer. Scaling makes the magnitudes well-behaved across architectures. Softmax turns the scores into weights that the model uses to mix the value vectors.