Transformers, From the Ground Up

The transformer architecture built one piece at a time: tokenization and embeddings, positional encoding, scaled dot-product attention, multi-head, residuals and layer norm, the feed-forward block, the full stack, training, and inference. Every step worked with tiny numeric matrices and interactive widgets.

1. 01Tokens and EmbeddingsA neural network cannot read letters. Day one turns text into the numbers a transformer actually consumes.7 sections
2. 02Positional EncodingAttention treats a sentence as a bag of words. Day two gives every position a fingerprint so word order survives.7 sections
3. 03AttentionAttention lets every token look at every other token and decide what to borrow. Today we compute one full pass by hand.8 sections
4. 04Multi-Head AttentionWhy one attention pass is not enough, and how splitting into heads lets a layer track several relationships at once.6 sections
5. 05Residuals and LayerNormThe two small tricks that let a transformer stack dozens of layers deep without the signal fading or the numbers exploding.7 sections
6. 06Feed-Forward NetworkAfter attention mixes information across tokens, a small two-layer network reshapes each token on its own. This is where most of a transformer's parameters live.7 sections
7. 07The Full ArchitectureEverything from days 1–6 clicks together into one stackable block — then you count the parameters of a real model.8 sections
8. 08TrainingHow a stack of random weights becomes a language model: one loss, one gradient, one careful step at a time.8 sections
9. 09Inference and DecodingA trained model only outputs probabilities. Turning them into text — fast — is its own craft.7 sections