Residuals and LayerNorm

A transformer is many attention-plus-MLP blocks stacked on top of each other. Day 04 built one attention block; a real model chains 12, 48, or more.

Stacking naively is dangerous. If each block multiplies the signal by a number a little below 1, then after 40 blocks the signal (and, during training, the gradient that flows back) has shrunk toward zero — this is the vanishing signal problem. If each block multiplies by a bit more than 1, the numbers explode instead. Either way, deep stacks refuse to train.

Two cheap fixes make deep stacks stable: a residual connection around every block, and LayerNorm to keep each vector at a sane scale.

A residual connection means a block does not replace its input — it adds to it. Instead of x -> Sublayer(x), the block computes x -> x + Sublayer(x). The original x always survives.

Take x = [2, 4, 6] and suppose the sublayer produces Sublayer(x) = [0.5, -1, 0.5]. The residual output is the elementwise sum:

Because the input is added straight through, there is always a clean path with a multiplier of exactly 1 from the output back to the input. Gradients ride that path unharmed, so even the 40th block still receives a strong training signal. The stack of these running sums is called the residual stream — think of it as a highway that every block reads from and writes a small correction onto.

LayerNorm keeps each vector at a stable scale before a block reads it, so no single dimension is allowed to blow up. It works on one vector at a time, across that vector's own features. Take x = [2, 4, 6].

Step 1 — mean: μ = (2 + 4 + 6) / 3 = 4.

Step 2 — variance: average the squared gaps from the mean.

σ² = (4 + 0 + 4) / 3 = 2.667, so σ = √2.667 ≈ 1.633 (a tiny ε is added inside the square root to avoid dividing by zero; here it is negligible).

Step 3 — normalize: x̂ = (x - μ) / σ.

Now x̂ = [-1.225, 0, 1.225] has mean 0 and variance 1.

Step 4 — scale and shift: the model does not have to keep mean 0 forever, so it learns a per-feature scale γ and shift β and computes y = γ ⊙ x̂ + β (elementwise). With γ = [1, 2, 0.5] and β = [0, 1, -1]:

The final y = [-1.225, 1.000, -0.388]. LayerNorm's only learned numbers are γ and β, one pair per feature — 2 · d_model parameters in all (here 6). It is cheap and it never lets a vector's scale drift.

Where you place LayerNorm relative to the residual add matters. The original transformer used post-LN: normalize after adding.

post-LN:  y = LayerNorm(x + Sublayer(x))
pre-LN:   y = x + Sublayer(LayerNorm(x))

Post-LN normalizes the whole residual stream every block, which can make very deep stacks touchy to train (they often need a careful learning-rate warmup). Pre-LN normalizes only the block's input and leaves the residual highway untouched, so the clean add-1 path from Day 05's residual section runs end to end. That extra stability is why modern models (GPT-2 onward) almost all use pre-LN.

• Deep stacks die from vanishing or exploding signal; residuals and LayerNorm fix both cheaply.
• A residual connection adds the input back (x + Sublayer(x)), giving gradients a clean path with multiplier 1.
• LayerNorm rescales one vector to mean 0 and variance 1, then applies learned γ (scale) and β (shift).
• Pre-LN (normalize the input) trains more stably than post-LN (normalize after the add) and is the modern default.

• Can you compute the mean and variance of a 3-vector by hand?
• Can you normalize that vector and then apply γ and β?
• Can you explain how a residual connection keeps gradients from vanishing?
• Can you state the difference between pre-LN and post-LN and say which modern models use?