Multi-Head Attention

On Day 03 you built a single attention pass: scores from Q·Kᵀ, scaled by √d_k, softmaxed into weights, then a weighted sum of the value vectors. That is exactly one head.

The problem: one head can only average the values one way. But language has many kinds of relationship at once. In "the cat that chased the mouse slept," a word needs to look back at its subject, at nearby adjectives, and at the verb it belongs to. A single averaging pass cannot serve all three at the same time.

Multi-head attention runs several of these passes in parallel. Each head gets its own slice of the query, key, and value space, so each head is free to learn a different relationship. One head might learn to look at the previous word, another at the sentence's subject. They never share weights, so they never collapse into the same behaviour.

Here is the whole mechanism with tiny numbers. We use d_model = 4 and 2 heads, so each head has width d_k = 2. Take two tokens whose projected Q, K, V rows are already computed (each is 4 wide):

Split. Head 1 takes columns 1-2, head 2 takes columns 3-4. Splitting only re-groups existing numbers; it creates nothing new.

Attend (head 1). Run the Day 03 recipe with d_k = 2, so √d_k = √2 ≈ 1.414. For the t1 query the raw scores are Q·Kᵀ: against K-t1 [1,0]·[1,0] = 1, against K-t2 [1,0]·[0,1] = 0. Scale by 1/1.414, then softmax.

For head1-t1: 0.670·[2,0] + 0.330·[0,2] = [1.340, 0.660].

Attend (head 2). Same recipe on head 2's slices. Here both value rows are [1,1], so every weighted sum lands on [1,1] — a reminder that the output depends on the values a head attends over.

Concat. For each token, glue head 1's output next to head 2's output to rebuild a 4-wide vector:

W_O. The concatenated vector is mixed by one more matrix, W_O (4×4), so the heads can talk to each other before leaving the layer. Using

W_O rows:
[1, 0, 1, 0]
[0, 1, 0, 1]
[1, 0, 0, 1]
[0, 1, 1, 0]

each output is concat · W_O. For t1: out[0] = 1.340·1 + 0.660·0 + 1·1 + 1·0 = 2.340, and continuing across the columns:

That 4-wide result is the multi-head attention output for each token — same width as the input, ready to add back into the residual stream (Day 05).

A common surprise: splitting into more heads does not add parameters. The projections W_Q, W_K, W_V are each d_model × d_model whether you use 1 head or 8 — the split is just a reshape of their output. Only four weight matrices carry parameters.

The general rule is 4 · d_model² weights for a multi-head attention block (ignoring biases). For our d_model = 4 that is 4 · 16 = 64. Doubling the heads from 2 to 4 while keeping d_model = 4 would leave this count unchanged — you would just have four heads of width 1 instead of two of width 2.

• Multi-head attention runs several independent attention passes so a layer can track several relationships at once.
• The flow is split -> attend per head -> concat -> W_O mix, and the output keeps the original d_model width.
• Splitting is a reshape, not new math: heads carve up the existing projected vectors.
• Parameters come only from W_Q, W_K, W_V, W_O — about 4 · d_model² weights, independent of head count.

• Can you split a 4-dim Q/K/V row into two 2-dim heads by hand?
• Can you run the Day 03 attention recipe separately inside each head?
• Can you concatenate two head outputs and apply W_O to get the final vector?
• Can you explain why adding heads (at fixed d_model) does not change the parameter count?