Feed-Forward Network

Attention (Days 03-04) moves information between tokens: each token gathers context from the others. But it does little to transform a single token's vector on its own — it mostly averages value vectors.

The feed-forward network fills that gap. It runs on each token separately (the same weights for every position, so it is called position-wise) and gives the model room to compute nonlinear features from the mixed-in context. A block alternates the two: attention to share, FFN to think. The FFN is just a small MLP: FFN(x) = W2 · GELU(W1 · x + b1) + b2.

The trick is to expand into a much wider hidden layer, apply a nonlinearity there, then project back to the original width. Real transformers expand 4x (d_ff = 4 · d_model); we use a tiny 2 -> 4 -> 2 here so the arithmetic stays readable.

Take an input token x = [1, 2] (d_model = 2).

Expand. W1 is 4×2 (one row per hidden neuron); with b1 = [0, 0, 0, 0] the pre-activation is h_pre = W1 · x.

W1 rows:
[1,  0]
[0,  1]
[1,  1]
[1, -1]

So h_pre = [1, 2, 3, -1] — the 2-wide token now lives in a 4-wide hidden space.

Activate. Apply GELU elementwise (see the next section for the shape):

So h = [0.841, 1.955, 2.996, -0.159].

Project back. W2 is 2×4 and maps the hidden vector back to width 2; with b2 = [0, 0], out = W2 · h.

W2 rows:
[1, 0, 1, 0]
[0, 1, 0, 1]

The FFN output is [3.837, 1.796] — back to the original d_model = 2, ready to add into the residual stream.

GELU (Gaussian Error Linear Unit) is the smooth curve the FFN uses instead of a hard ReLU. It is defined as GELU(z) = z · Φ(z), where Φ(z) is the probability that a standard normal draw is below z. In plain terms: it passes large positive values through almost unchanged, sends large negatives to nearly zero, and bends gently in between with a small negative dip.

Reading down the table you can see the shape: flat and near zero on the left, a shallow dip around z = -1, then rising toward the line y = z on the right. Unlike ReLU's sharp corner at 0, GELU is smooth everywhere, which helps gradients flow.

The FFN is small in idea but large in count, because of the 4x expansion. For our 2 -> 4 -> 2 block:

The weight count is about 2 · d_model · d_ff. With the real 4x rule (d_ff = 4 · d_model) that becomes roughly 8 · d_model² — larger than the attention block's 4 · d_model² from Day 04. In a real transformer block the FFN holds close to two-thirds of the parameters, even though attention gets most of the attention.

• The FFN is a position-wise two-layer MLP: expand with W1, apply GELU, project back with W2.
• It transforms each token on its own, complementing attention's cross-token mixing.
• GELU is a smooth activation: near zero for negatives, near identity for large positives.
• With the 4x expansion the FFN holds roughly two-thirds of a block's parameters (~8 · d_model²).

• Can you run a 2 -> 4 -> 2 forward pass by hand, including the GELU step?
• Can you describe the shape of the GELU curve from its value table?
• Can you explain why the FFN runs per token while attention runs across tokens?
• Can you count an FFN's parameters given d_model and d_ff?