Positional Encoding

On Day 01 each token became an embedding vector. But a plain stack of those vectors carries no sense of which came first. The attention step you will meet on Day 03 is order-invariant: if you shuffle the token vectors, it produces the same set of outputs, just shuffled. That is a real problem, because dog bites man and man bites dog share the exact same tokens and would look identical to the model.

We need to stamp each position with its own signature and fold that into the token's vector, so that "the same word in slot 0" and "the same word in slot 3" are no longer identical inputs.

There are two common ways to build that signature. A learned positional encoding keeps a second table, one trainable row per position, and trains it like any other weights. It is simple but caps the length: a model with 512 position rows cannot place a 513th token.

A sinusoidal encoding instead computes each position's vector from fixed sine and cosine waves — no training and no length cap, since the formula works for any position. Because it is the classic transformer choice and easy to check by hand, we will work it out in full.

The sinusoidal encoding fills the vector with waves of different frequencies. For a model dimension of 4, we pair up the slots: dims 0 and 1 form the first (fastest) pair, dims 2 and 3 the next (slower) pair. Each pair uses a sine in the even slot and a cosine in the odd slot. In plain terms:

dim 0 = sin(pos / 1)     dim 1 = cos(pos / 1)
dim 2 = sin(pos / 100)   dim 3 = cos(pos / 100)

The divisor grows for later pairs (1, then 100), which stretches the wave so it changes more slowly across positions. Plugging in positions 0 through 3 and rounding to three decimals:

Read one row as one position's fingerprint. Notice the fast pair (dims 0 and 1) swings a lot from row to row, while the slow pair (dims 2 and 3) barely moves. Two speeds together let close positions look similar and far positions look distinct.

Working one cell by hand: for position 1, dim 0 is sin(1) = 0.841 and dim 1 is cos(1) = 0.540. For dim 2 the argument is 1 / 100 = 0.01, and sin(0.01) = 0.010. No training was needed — every number came straight from the formula.

The last step is simple: add the position vector to the token's embedding, element by element. Suppose the token run had embedding [0.3, 0.1, 0.7, 0.2] (a 4-dim version) and it sits at position 1. Add row 1 of the table:

embedding    = [0.300, 0.100, 0.700, 0.200]
position 1   = [0.841, 0.540, 0.010, 1.000]
sum (input)  = [1.141, 0.640, 0.710, 1.200]

That sum is what actually enters the first transformer block. The same word at a different position gets a different sum, so order is now baked into the numbers. We add rather than concatenate so the vector length stays the same and no extra slots are spent.

• Attention is order-invariant, so raw token vectors lose word order.
• A positional encoding gives each position a distinct vector.
• Learned encodings train one row per position but cap the sequence length.
• Sinusoidal encodings compute positions from sine/cosine waves, uncapped and untrained.
• The position vector is added to the embedding, so order enters the model for free.

• [ ] Can you explain why dog bites man and man bites dog confuse an order-blind model?
• [ ] Can you fill one row of the sin/cos table for a given position?
• [ ] Can you say why later dimension-pairs use a larger divisor?
• [ ] Can you add a position vector to an embedding to get the block's input?