Transformers, From the Ground Up

The transformer architecture built one piece at a time: tokenization and embeddings, positional encoding, scaled dot-product attention, multi-head, residuals and layer norm, the feed-forward block, the full stack, training, and inference. Every step worked with tiny numeric matrices and interactive widgets.

1. 01Tokens and EmbeddingsA neural network cannot read letters. Day one turns text into the numbers a transformer actually consumes.Published 04 Jul 20267 sections
2. 02Positional EncodingAttention treats a sentence as a bag of words. Day two gives every position a fingerprint so word order survives.Published 04 Jul 20267 sections
3. 03AttentionAttention lets every token look at every other token and decide what to borrow. Today we compute one full pass by hand.Published 04 Jul 20268 sections
4. 04Multi-Head AttentionWhy one attention pass is not enough, and how splitting into heads lets a layer track several relationships at once.Published 04 Jul 20266 sections
5. 05Residuals and LayerNormThe two small tricks that let a transformer stack dozens of layers deep without the signal fading or the numbers exploding.Published 04 Jul 20267 sections
6. 06Feed-Forward NetworkAfter attention mixes information across tokens, a small two-layer network reshapes each token on its own. This is where most of a transformer's parameters live.Published 04 Jul 20267 sections
7. 07The Full ArchitectureEverything from days 1–6 clicks together into one stackable block — then you count the parameters of a real model.Published 04 Jul 20268 sections
8. 08TrainingHow a stack of random weights becomes a language model: one loss, one gradient, one careful step at a time.Published 04 Jul 20268 sections
9. 09Inference and DecodingA trained model only outputs probabilities. Turning them into text — fast — is its own craft.Published 04 Jul 20267 sections