RLHF and Alignment
Fine-tuning teaches the model to imitate good answers; alignment teaches it which answer people actually prefer.
Key terms
| Term | Plain definition |
|---|---|
| Alignment | Shaping a model so its outputs match human intentions and values, not just the training text. |
| RLHF | Reinforcement Learning from Human Feedback — using human preferences as the training signal. |
| Preference pair | Two model responses to the same prompt, with a human label saying which is better. |
| Reward model | A model that reads a response and outputs a single score for how good it is. |
| Policy | The language model being trained; it chooses (a "policy" over) the next tokens. |
| PPO | Proximal Policy Optimization — the RL algorithm often used to push the policy toward higher reward. |
| KL penalty | A leash that keeps the tuned policy close to the SFT model so it does not drift into nonsense. |
| DPO | Direct Preference Optimization — a shortcut that skips the reward model and RL loop. |
Why SFT is not enough
Yesterday's supervised fine-tuning (Day 4) teaches the model to imitate ideal answers. But writing an "ideal" answer for every prompt is hard, and there is rarely one right answer — a good reply can be more or less helpful, more or less safe. Imitation cannot capture "this reply is better than that one."
Humans, though, are good at comparing. Show a person two responses and they can usually pick the better one even when they could not write the perfect one themselves. RLHF turns those comparisons into a training signal.
From preference data to a reward model
We start with the SFT model and collect preference pairs: for a prompt, sample two responses and ask a human which they prefer. From many such judgments we train a reward model that assigns each response a single score, tuned so preferred responses score higher than rejected ones.
Take the prompt "Explain photosynthesis to a child" and three sampled responses:
| Response | Style | Reward r |
|---|---|---|
| R1 | Clear, warm, uses a simple analogy | 2.0 |
| R2 | Correct but dry and technical | 0.5 |
| R3 | Vague and partly wrong | -1.0 |
The reward model is trained so the probability a human prefers one response over another follows sigmoid(r_winner - r_loser). Plugging in the scores above:
| Comparison | r gap | sigmoid(gap) = predicted preference |
|---|---|---|
| R1 over R2 | 2.0 - 0.5 = 1.5 | 1 / (1 + e^-1.5) = 0.82 |
| R1 over R3 | 2.0 - (-1.0) = 3.0 | 1 / (1 + e^-3.0) = 0.95 |
| R2 over R3 | 0.5 - (-1.0) = 1.5 | 1 / (1 + e^-1.5) = 0.82 |
A bigger reward gap means a more confident preference. When the gap is 0 the model predicts a 50/50 toss-up — it has no opinion.
The RL step (a PPO sketch)
With a reward model in hand we improve the policy (the language model). One round of PPO looks like this: sample a response from the current policy, score it with the reward model, then adjust the weights to make high-reward responses more likely and low-reward ones less likely.
Left unchecked, the policy would learn to game the reward model — spitting out weird text that happens to score high. So PPO adds a KL penalty: a term that punishes the policy for straying too far from the SFT model. The effective objective is roughly "maximize reward, minus a penalty for drifting." The model climbs toward higher reward while staying fluent.
The whole loop, end to end:
The DPO shortcut
RLHF has moving parts: a reward model, a sampling loop, an RL algorithm, and a KL leash — all of which can be finicky to tune. Direct Preference Optimization (DPO) collapses them.
DPO skips training a separate reward model and skips the RL loop entirely. It rearranges the math so the same preference pairs become a single classification-style loss on the policy directly: push up the probability of each preferred response and push down the rejected one, with a built-in term that plays the role of the KL leash. Same preference data, far simpler pipeline — which is why DPO is now a common default.
Key takeaways
- SFT imitates ideal answers; alignment learns which answer people prefer from comparisons.
- A reward model turns preference pairs into a scalar score per response.
- Predicted preference follows
sigmoid(r_winner - r_loser)— bigger gap, more confident. - PPO pushes the policy toward higher reward while a KL penalty keeps it close to the SFT model.
- DPO reaches a similar result by turning preferences into one direct loss, skipping the reward model and RL loop.
Checklist
- [ ] Can you explain why comparing two answers is easier for humans than writing the perfect one?
- [ ] Can you compute a predicted preference from two rewards using
sigmoid(r_winner - r_loser)? - [ ] Can you say what the KL penalty prevents and why it matters?
- [ ] Can you name the four stages of the RLHF pipeline in order?
- [ ] Can you state what DPO removes compared with full RLHF?