Attention

Attention is a soft lookup. Each token produces three vectors: a query (what it wants), a key (what it offers), and a value (what it hands over). A token compares its query against every token's key to see who is relevant, then collects a blend of their values weighted by relevance.

Think of a library. Your query is the topic in your head; each book's key is its spine label; the value is the book's actual contents. You match your query to the labels, then read mostly from the best matches. We will run this end to end for three tokens, each with 2-dimensional vectors so every number is checkable.

Our three tokens carry these vectors:

We will compute what token 1 attends to, using its query q1 = [1, 0].

A score measures how well token 1's query matches each token's key. The measure is the dot product q · k: multiply matching slots and add. For q1 = [1, 0] against each key:

score(1,1) = q1 · k1 = 1*1 + 0*0 = 1
score(1,2) = q1 · k2 = 1*0 + 0*1 = 0
score(1,3) = q1 · k3 = 1*1 + 0*1 = 1

So token 1's raw scores are [1, 0, 1]. A big score means the query and key point the same way; a zero means they are unrelated. Doing this for all three queries at once is the matrix product Q·Kᵀ — the transpose ᵀ just lines the keys up as columns so every query meets every key.

Dot products grow with dimension: add more slots and the sums get larger, which pushes softmax into sharp, extreme weights that are hard to train. To keep scores in a calm range we divide every score by √d_k, the square root of the query/key dimension. Here d_k = 2, so we divide by √2 ≈ 1.414:

1 / 1.414 = 0.707
0 / 1.414 = 0.000
1 / 1.414 = 0.707

The scaled scores are [0.707, 0.000, 0.707]. Same shape as before, gentler size.

Raw scores can be any size and do not add up to anything meaningful. Softmax fixes both: it exponentiates each score (making all values positive) and then divides by the total, so the results are positive and sum to exactly 1. Step one, take exp of each scaled score:

exp(0.707) = 2.028
exp(0.000) = 1.000
exp(0.707) = 2.028
sum        = 2.028 + 1.000 + 2.028 = 5.056

Step two, divide each by the sum to normalize:

w1 = 2.028 / 5.056 = 0.401
w2 = 1.000 / 5.056 = 0.198
w3 = 2.028 / 5.056 = 0.401
check: 0.401 + 0.198 + 0.401 = 1.000

These attention weights [0.401, 0.198, 0.401] say token 1 leans on tokens 1 and 3, and less on token 2. The widget below lets you pick which token is the query and watch its scores turn into a full weight grid.

The output for token 1 is the blend of every token's value, weighted by those attention weights. Multiply each value vector by its weight and add them:

0.401 * v1 = 0.401 * [1, 0] = [0.401, 0.000]
0.198 * v2 = 0.198 * [0, 1] = [0.000, 0.198]
0.401 * v3 = 0.401 * [1, 1] = [0.401, 0.401]

Add the three, slot by slot:

dim 0: 0.401 + 0.000 + 0.401 = 0.802
dim 1: 0.000 + 0.198 + 0.401 = 0.599
output(token 1) = [0.802, 0.599]

That vector is token 1's attention output: a mix pulled mostly from tokens 1 and 3, exactly as the weights predicted. Run the same three steps with q2 and q3 and you have the full attention layer — scores, scale by √d_k, softmax, weighted sum.

• Every token emits a query, a key, and a value.
• Scores are dot products of one query against every key (Q·Kᵀ).
• Dividing by √d_k keeps scores calm so softmax stays trainable.
• Softmax exponentiates then normalizes, giving weights that sum to 1.
• The output is the values blended by those weights.

• [ ] Can you compute the three scores for a given query and set of keys?
• [ ] Can you say why we divide the scores by √d_k?
• [ ] Can you run softmax on a row of scores: exponentiate, sum, divide?
• [ ] Can you produce the output vector as a weighted sum of the value vectors?