Pretraining and Perplexity
The whole of pretraining is one stubborn game: guess the next token, measure the surprise, nudge, repeat — a trillion times.
Key terms
| Term | Meaning |
|---|---|
| Token | A chunk of text (word, sub-word, or byte) the model reads and predicts |
| Pretraining | Training on raw text to predict the next token, with no task labels |
| Next-token prediction | For each position, output a probability for every possible next token |
| Surprise | How unexpected the true next token was: -log2(p), measured in bits |
| Cross-entropy | The average surprise over all predictions — the loss pretraining minimizes |
| Perplexity | 2^H, the "effective number of choices" the model faces per token |
| Bit | The unit of surprise when we take logarithms in base 2 |
Next-token prediction as the whole game
Before anything else, know what the model is actually trained to do. It reads a stretch of text left to right and, at every position, outputs a probability for each token that could come next. Training compares that guess to the token that actually came next and nudges the weights so the correct token gets a little more probability. There is no other objective — no labels, no questions, no answers. Everything an LLM later appears to "know" is a side effect of getting very good at this one prediction, repeated over trillions of tokens.
Surprise and cross-entropy
To improve the guess we need a number that says how wrong it was. That number is
surprise: -log2(p), where p is the probability the model gave to the
token that truly appeared. A confident, correct guess (p near 1) costs almost
no surprise; a rare true token the model doubted (p near 0) is very
surprising. Cross-entropy is just the average surprise across every
prediction — the exact loss pretraining pushes downward.
Work it out on a tiny 4-token sequence. At each step the model gave the true
next token this probability, so its surprise is -log2(p):
| Step | True token | Model's p | Surprise -log2(p) (bits) |
|---|---|---|---|
| 1 | cat | 0.5 | 1.0 |
| 2 | sat | 0.25 | 2.0 |
| 3 | on | 0.25 | 2.0 |
| 4 | mat | 0.5 | 1.0 |
Cross-entropy is the average of that last column:
H = (1.0 + 2.0 + 2.0 + 1.0) / 4 = 6.0 / 4 = 1.5 bits per token
The whole point is that lower cross-entropy means the model was, on average, less surprised by real text — which is exactly what "predicts well" means. Drag the slider below to feel how one token's probability turns into its surprise in bits.
Perplexity = 2^H
Cross-entropy in bits is precise but not intuitive. Perplexity converts it
back into a count of choices: PP = 2^H. Read it as "the model is as confused
as if it were picking uniformly among this many equally-likely tokens." A
perplexity of 1 is a perfect oracle; a perplexity equal to the vocabulary size
is pure guessing.
Take the H = 1.5 bits we just computed and raise 2 to it. Split the exponent
into a whole part and a half:
| Quantity | Value |
|---|---|
H (cross-entropy) | 1.5 bits |
2^1 | 2 |
2^0.5 (that is √2) | ≈ 1.414 |
PP = 2^1.5 = 2^1 · 2^0.5 | ≈ 2.828 |
So this toy model is effectively choosing among about 2.83 tokens at each step — far better than blind guessing over a full vocabulary, but nowhere near certain. Every halving of perplexity means one fewer bit of average surprise.
What the data is
Now that the objective and its yardstick are clear, the last piece is what text the model reads. Pretraining corpora are enormous mixes of web pages, books, code, and reference text, then heavily cleaned: near-duplicate pages removed, low-quality or unsafe text filtered out, and everything cut into tokens (sub-word pieces), not whole words. The model usually sees each token roughly once. Nothing here is labeled — the "answer" at every position is simply the token that happens to come next in the cleaned text.
Key takeaways
- Pretraining optimizes one objective: predict the next token's probability.
- Surprise
-log2(p)measures a single guess; cross-entropy averages it — that average is the loss. - Perplexity
2^Hreads cross-entropy as an effective number of choices per token. - For our toy sequence:
H = 1.5 bits, soPP = 2^1.5 ≈ 2.828. - The data is huge, deduped, filtered text cut into tokens, with no labels.
Checklist
- [ ] Can you state the single objective of pretraining in one sentence?
- [ ] Can you compute the surprise of a guess given the true token's probability?
- [ ] Can you turn a list of per-step surprises into a cross-entropy?
- [ ] Can you convert a cross-entropy in bits into a perplexity, and say what it means?