Fine-Tuning and LoRA

Pretraining teaches a model language; fine-tuning teaches it a job — and LoRA does it without touching most of the weights.

Key terms

Term	Plain definition
Pretrained model	The base model from Day 1 — good at predicting the next token, but not shaped to follow instructions.
Fine-tuning	Continuing training on a smaller, task-specific dataset so the model adopts a new behavior.
Supervised fine-tuning (SFT)	Fine-tuning on labeled `(prompt, ideal answer)` pairs so the model imitates good responses.
Full fine-tuning	Updating every weight in the model — accurate but expensive to store and run.
LoRA	Low-Rank Adaptation: freeze the original weights and learn a tiny low-rank update instead.
Rank	The number `r` of independent directions in a matrix; a low-rank update uses a small `r`.
Adapter	A small set of extra weights added to a frozen model to specialize it.

Pretrained models are not yet helpful

A model fresh from pretraining (Day 1) is a next-token machine. Ask it a question and it may continue with more questions, because internet text often lists questions together. It has the knowledge but not the habit of answering.

Fine-tuning fixes the habit. We show the model many examples of the behavior we want — answering, summarizing, following an instruction — and nudge its weights toward producing that behavior. The knowledge from pretraining stays; only the response style shifts.

Supervised fine-tuning

Supervised fine-tuning (SFT) is the simplest recipe. We collect a dataset of (prompt, ideal answer) pairs written or curated by humans, then train the model with the same next-token objective from Day 1 — but now the "next tokens" are the ideal answer.

The loss is unchanged: average surprise over the answer tokens (Day 1). What changes is the data. After a few thousand well-chosen pairs, the model learns to treat a prompt as something to respond to rather than merely continue.

The catch: full fine-tuning updates every weight. A 7-billion-parameter model then needs a second 7-billion-parameter copy stored for each task. That is where LoRA comes in.

LoRA: freeze W, learn B times A

LoRA keeps the original weight matrix W frozen and learns a small update ΔW that is added at run time: W_effective = W + B·A. Here B and A are two skinny matrices whose product has the same shape as W, but far fewer parameters.

Take a tiny weight matrix W of shape 4 by 4 — that is 4·4 = 16 numbers. Instead of retraining all 16, LoRA learns a rank-1 update: B of shape 4 by 1 and A of shape 1 by 4.

W (frozen, 16 numbers):

	c1	c2	c3	c4
r1	0.2	0.1	0.0	0.3
r2	0.1	0.4	0.2	0.1
r3	0.0	0.2	0.5	0.0
r4	0.3	0.1	0.0	0.2

The learned factors — B (4 by 1) and A (1 by 4):

| B | | --- | | 1 | | 2 | | 0 | | 1 |

A →	a1	a2	a3	a4
	1	0	1	2

The update ΔW = B·A fills each cell with B[row] · A[col]:

ΔW	c1	c3	c4
r1 (B=1)	1	1	2
r2 (B=2)	2	2	4
r3 (B=0)	0	0	0
r4 (B=1)	1	1	2

Row 2 is just row 1 doubled, and row 4 equals row 1 — every row is a multiple of the single pattern A. That is exactly what "rank 1" means: one direction, scaled per row. At run time the model uses W + ΔW.

Why low rank is enough — count the parameters

The payoff is the parameter count. Compare storing a full update against storing the two LoRA factors:

Update	Numbers to learn	Formula
Full 4 by 4	16	`d · d`
LoRA rank 1	8	`d·r + r·d = 2·d·r`

For this tiny d = 4, LoRA already halves the count: 2·4·1 = 8 versus 16. The gap explodes at real sizes. For a d = 4096 weight matrix:

Update	Numbers to learn
Full (`4096·4096`)	16,777,216
LoRA rank 8 (`2·4096·8`)	65,536

That is roughly 0.4% of the full count — for one matrix. Why can such a thin update work at all? Fine-tuning does not need to rebuild the model; it only needs to steer it. Empirically the adjustment a task requires lives in a small number of directions, so a low-rank B·A captures most of the benefit while the frozen W keeps all the pretrained knowledge. You can even keep many small LoRA adapters — one per task — and swap them on top of a single frozen base.

Key takeaways

Pretraining gives knowledge; fine-tuning gives the behavior (answering, following instructions).
SFT reuses the Day 1 next-token loss, but on curated (prompt, ideal answer) pairs.
Full fine-tuning updates every weight and needs a full model copy per task.
LoRA freezes W and learns a low-rank update B·A, storing a tiny fraction of the parameters.
Low rank works because a task's needed adjustment lives in a few directions; adapters can be swapped per task.

Checklist

[ ] Can you explain why a pretrained model answers a question with more questions?
[ ] Can you state what SFT changes versus what it keeps from pretraining?
[ ] Can you compute B·A for a 4 by 1 and 1 by 4 pair and get a 4 by 4 update?
[ ] Can you compare the parameter count of a full update versus LoRA using d·d versus 2·d·r?
[ ] Can you say in one sentence why a low-rank update is usually enough?