Scaling Laws
Bigger is not a hope, it is a curve. Scaling laws let you predict a model's loss before you spend a dollar training it.
Key terms
| Term | Meaning |
|---|---|
| Scaling law | A power-law relationship between loss and a resource (params, data, or compute) |
| Power law | L ≈ a · x^(-b): loss falls by a fixed fraction each time x multiplies |
| N (parameters) | The number of weights in the model |
| D (tokens) | The number of training tokens the model sees |
| C (compute) | Total training work in FLOPs, roughly C ≈ 6 · N · D |
| Compute-optimal | The N and D that give the lowest loss for a fixed compute budget |
| Chinchilla | The finding that optimal training uses about 20 tokens per parameter |
Loss follows power laws
Yesterday we measured a model with cross-entropy loss. The first surprise of
scaling research is that this loss is not random from model to model — it falls
along a smooth power law as you add resources. Written out, L ≈ a · x^(-b),
where x is model size, data, or compute. The tell-tale sign of a power law is
that it becomes a straight line when both axes are logarithmic, because
multiplying x by 10 subtracts a fixed amount from the loss rather than a
fixed fraction of what remains.
That straight-line behavior is the whole intuition — every 10x of compute buys roughly the same drop in loss, so gains are real but never free:
log loss | +. | '. | '. | '. each 10x compute | '. -> same size drop | '. | '. +----------------+-- log compute
Params vs data vs compute
A power law in what, exactly? There are three knobs, and they are linked. N is the parameter count, D is the number of training tokens, and C is the total compute. A good rule of thumb ties them together: one forward-and-backward pass costs about 6 FLOPs per parameter per token, so
C ≈ 6 · N · D
This is the key constraint. If your compute budget C is fixed, then spending it
on a bigger model (larger N) leaves fewer tokens (smaller D), and vice
versa. Scaling laws answer the trade-off: for a fixed C, which split of N and
D reaches the lowest loss?
Chinchilla (≈20 tokens/param)
The Chinchilla result gave a crisp answer: to be compute-optimal, scale data
and parameters together at roughly 20 tokens per parameter (D ≈ 20 · N).
Many earlier models broke this rule — they were too big and trained on too few
tokens, leaving loss on the table. A model that obeys the ratio beats a much
larger, under-trained one at the same compute.
Here is the compute-optimal recipe across sizes, using D = 20·N and
C ≈ 6·N·D:
Parameters N | Optimal tokens D = 20·N | Compute C ≈ 6·N·D (FLOPs) |
|---|---|---|
| 125M | 2.5B | ≈ 1.9 × 10^18 |
| 1.3B | 26B | ≈ 2.0 × 10^20 |
| 7B | 140B | ≈ 5.9 × 10^21 |
| 70B | 1.4T | ≈ 5.9 × 10^23 |
Notice how tokens grow in lockstep with parameters — doubling the model without doubling the data is exactly the mistake Chinchilla warned against.
What it predicts
The practical payoff is planning. Because loss lies on a power law, you can fit the curve on a few small training runs and then read off, before committing to a huge run: how big a model to build, how many tokens to gather, and what loss to expect. It also sets honest expectations — the power-law exponent is small, so each further drop in loss costs an order of magnitude more compute. Scaling works, but it is a straight line on a log-log plot, not a cliff.
Key takeaways
- Loss falls as a power law in params, data, and compute — a straight line on log-log axes.
- The three knobs are tied by
C ≈ 6 · N · D. - Chinchilla: for a fixed compute budget, use about 20 tokens per parameter.
- Under-trained giant models waste compute; scale
NandDtogether. - Scaling laws let you predict optimal size, data, and loss before the full run.
Checklist
- [ ] Can you explain why a power law looks straight on a log-log plot?
- [ ] Can you estimate training compute from
NandDusingC ≈ 6·N·D? - [ ] Can you state the Chinchilla tokens-per-parameter rule of thumb?
- [ ] Can you say what a scaling law lets you predict before training?