Proving Adaptivity

The whole system so far rests on one number: mastery strength (Day 3), the per-child, per-concept measure of how well a child knows something. The frontier reads it, the tree reads it, the parents' map is colored by it. If that number is wrong, everything downstream is wrong — the adaptivity is theater. This day is about discovering the number was wrong, and why the way it was wrong is so easy to miss.

The symptom was subtle. The system ran, wrote data, showed progress — nothing crashed, no error, no 500. And yet, looking across children, something was off: every child ended up looking equally "mastered", regardless of what they actually did. A signal that is the same for everyone is not a signal at all. The bug was not a crash; it was a degenerate signal — technically alive, informationally dead.

The diagram shows what "degenerate" looks like: three very different children, one identical verdict. Nothing is broken in the "it threw an exception" sense — which is exactly why this class of bug survives tests that only check that a write happened.

To find the cause, recall Day 6: a byte is a sequence of bits, and only some bits are scored (they judge a right/wrong answer). The Arabic byte ended with a speak bit — the child says the sound aloud. The problem: the speak bit was being treated as if finishing it proved mastery. But a speak bit is not scored — there is no right/wrong. A child can simply tap through it: tap, move on, done.

So the mastery update was firing on an action that carried no evidence of learning. Every child who reached the end of the byte — whether they truly knew the sound or just tapped past the speak screen — got the same mastery bump. The number went up for everyone, equally, for doing nothing that demonstrated recall. That is how three different children collapse to one identical "mastered".

The diagram pinpoints the defect: the mastery increment was wired to completing the speak bit, which anyone can tap through, instead of to the scored recognize bit that actually tests recall. Fixing it means being precise about which kinds of bits are allowed to move mastery at all.

The fix was to classify every bit by its cognitive phase and route mastery updates by phase. The phases spell ERPA, and naming them is what made the rule statable:

• E — Expose: show the concept. The child sees it. No judgment.
• R — Recognize: the child picks it out / identifies it. Scored — there is a right answer.
• P — Produce: the child generates it (e.g. speaks it). Usually not scored.
• A — Apply: the child uses it in a new context. Scored — there is a right answer.

The routing rule, stated once and applied everywhere: only scored phases (Recognize, Apply) move real mastery. Exposure phases (Expose, Produce) grant a small, bounded exposure credit — and nothing more.

The diagram is the fix in one picture: two lanes. Expose and Produce feed a capped exposure credit (about +0.25 per session, and it cannot climb past a ceiling on its own); Recognize and Apply — the scored phases — are the only ones that move genuine mastery. So a child who only taps through the speak (Produce) bit accrues a little bounded exposure and stops; a child who actually answers the scored recognize bit moves real mastery. The signal differentiates again, because it is now tied to actions that carry evidence.

• Tap-through is capped. Exposure credit is bounded (+0.25/session, with a ceiling), so no amount of tapping through unscored bits can ever reach "mastered" on its own.
• Mastery requires evidence. Only a scored phase — where the child was actually right or wrong — moves the mastery number, so the number once again means "this child demonstrated recall."
• Different children now diverge. A child who answers correctly and a child who taps through end up in genuinely different places, which is the entire point of an adaptive system.

The most valuable thing that came out of this bug is a rule about how you test adaptivity, because the original tests were green the whole time. They asserted the wrong thing.

The failing tests checked that a mastery write happened — and it did happen, for everyone, which is precisely the bug. A test that only asserts "the system recorded progress" cannot catch a signal that records identical progress for everybody. The write occurring is not evidence the signal works.

The diagram contrasts the two assertions. The rule to carry away: an adaptivity test must assert output differentiation — that different inputs (a real answer vs a tap-through) produce different outputs — not merely that a write occurred. Any adaptive feature you build should be tested by feeding it two distinguishable learners and proving the system distinguishes them. If it can't, the adaptivity is decorative, no matter how green the "a write happened" test is.

This is also where Day 8's safety paid off: because the graph engine shipped behind a flag, fail-closed, to a beta cohort, this degenerate-signal bug was caught and fixed on a small, watched group — not discovered in production across every family. Safe shipping and honest signals are the two halves of the same discipline. Day 10 steps back to what the finished, corrected system does in the real world.

• An adaptive system is only as trustworthy as its mastery signal; a signal that reads the same for every child is degenerate — alive (no crash) but informationally dead.
• The root cause was crediting mastery for completing an unscored bit (the speak/Produce bit), which a child can tap through — so every child got the same bump for no evidence of recall.
• The fix, ERPA-phase routing, classifies bits as Expose / Recognize / Produce / Apply and routes updates by phase: only the scored phases (Recognize, Apply) move real mastery; Expose/Produce grant only a bounded exposure credit (~+0.25/session, capped).
• After the fix, tap-through can never reach "mastered", mastery again requires demonstrated recall, and different children's signals genuinely diverge.
• The durable lesson: an adaptivity test must assert output differentiation (different learners → different outputs), not merely that a write happened — the original green tests only checked the write, which is why they missed the bug.

• [ ] I can explain what a "degenerate signal" is and why it is dangerous despite no crash or error.
• [ ] I can trace the root cause: mastery credited for an unscored, tap-through-able bit.
• [ ] I can list the four ERPA phases and say which two are scored and allowed to move mastery.
• [ ] I can explain how bounded exposure credit (+0.25/session, capped) prevents tap-through from reaching mastery.
• [ ] I can state the testing lesson — assert output differentiation, not just that a write occurred — and design such a test.