Sampling

The simplest decoder is greedy: at every step take the token with the highest probability. It sounds safe, but it is brittle. Because the same context tends to produce the same top token, greedy output slides into loops — "the best way to do this is the best way to do this is..." — and never explores the many reasonable continuations sitting just below the top. Real text has variety, so we usually want to sample from the distribution rather than always take its peak. The three dials below control how that sampling behaves.

The first dial reshapes the whole distribution before we sample. Temperature T divides every logit before the softmax: p_i = softmax(z_i / T). A low T (below 1) exaggerates the gaps, making the top token even more dominant — closer to greedy. A high T (above 1) shrinks the gaps, spreading probability toward the long tail and making output more random.

Take five tokens with fixed logits z = [2, 1, 0, -1, -2] and run the softmax at three temperatures. Watch the top token's share fall as T rises:

At T=0.5 token A owns almost all the mass (0.865); at T=2 the distribution is much flatter (0.429) and rare tokens become reachable. Same model, same logits — only the temperature changed. Drag the temperature dial below and watch the bars and entropy readout reshape live.

Temperature reshapes every token's probability but never removes a token — even absurd tail options keep a tiny chance. Top-k truncates instead: keep only the k highest-probability tokens, drop the rest, and renormalize so the survivors sum to 1.

Using the T=1 column above with k=2, we keep A (0.636) and B (0.234) and discard the rest. Their sum is 0.636 + 0.234 = 0.870, so we divide each by 0.870 to renormalize:

Now only A and B can ever be sampled. Top-k is simple but rigid — k=2 is too few when many tokens are plausible and too many when one token should dominate.

Top-p fixes that rigidity by keeping a variable number of tokens. Sort by probability, walk down the list accumulating mass, and stop as soon as the running total reaches the threshold p. The kept set (the "nucleus") is large when the model is unsure and small when it is confident.

With the T=1 distribution and p = 0.9, walk the cumulative sum:

A: 0.636              -> cumulative 0.636  (below 0.9, keep going)
B: 0.234              -> cumulative 0.870  (below 0.9, keep going)
C: 0.086              -> cumulative 0.956  (>= 0.9, stop)

We keep A, B, C (their sum is 0.956) and renormalize by dividing each by 0.956:

Because the nucleus adapts to the model's confidence, top-p is the most common default. Move the top-k and top-p sliders below to dim the cut tokens, then hit Sample to draw one from what remains.

With all three dials understood, a few practical defaults follow. For factual or code output, stay near greedy: low temperature (around 0.2), which keeps answers tight and repeatable. For creative or open-ended text, raise temperature toward 0.8-1.0 and add top-p around 0.9 so the model explores without wandering into nonsense. Top-k and top-p can be combined, but top-p alone handles most cases because it adapts to how confident the model is at each step.

• Greedy decoding is repetitive; sampling from the distribution adds needed variety.
• Temperature T sharpens (T<1) or flattens (T>1) the softmax before sampling.
• Top-k keeps a fixed number of tokens; top-p keeps a variable nucleus by cumulative mass.
• After any cut you must renormalize so the kept probabilities sum to 1.
• Low temperature suits facts and code; higher temperature with top-p ≈ 0.9 suits creative text.

• [ ] Can you explain why greedy decoding tends to loop?
• [ ] Can you compute a softmax at a given temperature and say how T changes it?
• [ ] Can you apply top-k to a distribution and renormalize the survivors?
• [ ] Can you find the top-p nucleus for a threshold and renormalize it?