The Full Architecture

On days 3–6 you built the pieces separately: masked self-attention, LayerNorm, the residual add, and the feed-forward network. A transformer block is just those pieces in a fixed order, repeated. Nothing new is introduced here — you only wire them up.

A modern block is pre-LN: normalize first, run the sub-layer, then add the result back into the residual stream. Each sub-layer is a small detour off the residual highway; the highway itself is never overwritten, only added to.

In words: x -> LN -> attention -> add x, then -> LN -> FFN -> add again. The two add steps are the residual connections from day 5; the two LayerNorms are the normalizers from that same day. The attention detour mixes information across tokens; the FFN detour transforms each token on its own (day 6).

One block can only do so much. Depth comes from stacking many identical blocks: the output of block 1 becomes the input of block 2, and so on. Because every block reads and writes the same-width residual stream, they clip together like identical bricks.

The flow start-to-finish: turn token ids into vectors and add positions (days 1–2), push them through N blocks, apply one final LayerNorm, then project each position's vector to a score (logit) for every word in the vocabulary. GPT-2 small uses N = 12.

A language model must predict the next token from the tokens so far — it may never peek at the answer. Self-attention, left alone, lets every position see every other position, including future ones. The causal mask fixes that by blocking any position from attending to positions to its right.

Concretely, before the softmax (day 3) we set the score for every "future" key to negative infinity. Softmax turns those into weight 0. Here 1 marks an allowed attention edge and 0 a blocked one, for four tokens:

Token t1 attends only to itself; t4 attends to all four. The mask is a lower-triangular pattern. Mechanically, a blocked cell's raw score becomes -inf, so softmax gives it weight 0 and it contributes nothing to the weighted sum of values:

masked score -> -inf
softmax(-inf) -> 0
weight 0 => value ignored

The original 2017 Transformer had two halves: an encoder that reads the whole input and a decoder that writes the output. Most modern LLMs drop the encoder and keep only the masked decoder stack — the shape you just assembled.

Cross-attention is the one new idea in the encoder-decoder shape: the decoder runs an extra attention step whose queries come from the decoder but whose keys and values come from the encoder's output — letting the output attend back to the input. Decoder-only models skip it, which is why this course focuses on the GPT shape.

"124 million parameters" sounds mysterious until you add it up. GPT-2 small has d_model = 768, N = 12 blocks, a feed-forward width of d_ff = 3072 (the 4× expansion from day 6), a vocabulary of 50257, and a context length of 1024. Every parameter lives in one of a few buckets.

First, one block. The attention sub-layer has four 768 × 768 projections (Q, K, V, and the output projection W_O from day 4), each with a 768 bias. The FFN has the 768 -> 3072 and 3072 -> 768 weight matrices with biases. Two LayerNorms contribute a scale and shift per dimension.

Now the whole model. The two embedding tables and the final LayerNorm sit outside the blocks. GPT-2 ties the output projection to the token-embedding table (reuses the same weights), so the head adds nothing new.

That last number, 124,439,808, is the "124M" everyone quotes. Notice how much of it is the token embedding — nearly a third — simply because the vocabulary is large. Scaling up the model means mostly growing d_model, d_ff, and N, which quadruple the block cost.

• A transformer block is days 3–6 wired in a fixed order: pre-LN, attention, residual add, pre-LN, FFN, residual add.
• Depth is just N identical blocks reading and writing the same-width residual stream.
• The causal mask sets future-token scores to -inf so softmax gives them weight 0.
• Decoder-only (GPT) keeps only the masked stack; encoder-decoder adds cross-attention to the input.
• GPT-2 small = 12 blocks + embeddings + final LayerNorm = 124,439,808 parameters, and the arithmetic is fully checkable.

• [ ] Can you draw one pre-LN block from memory, marking both residual adds?
• [ ] Can you explain why blocks stack cleanly (same-width residual stream)?
• [ ] Can you fill in a 4×4 causal mask and say what softmax does to a -inf score?
• [ ] Can you state the one new operation an encoder-decoder adds (cross-attention)?
• [ ] Can you reproduce the 124M parameter count bucket by bucket?