Close the Generation–Verification Gap
Source: Stanford CS329A, “Test-Time Compute Scaling,” 00:04:22–00:04:32 and 00:13:12–00:22:00 — https://www.youtube.com/watch?v=-Ggc37xLj_Y&t=792s • Course context: https://cs329a.stanford.edu/
More candidates need a trustworthy selector
Day 02 handed you candidate-corpus.json plus difficulty-profile.json. Today you will calibrate selectors without exposing protected labels, then emit selector-calibration.json. The generation–verification gap is the difference between an analysis oracle finding a correct answer somewhere and a deployable selector identifying it. Stanford’s lecture uses unit tests as an example of automated selection at 00:04:22–00:04:32.
Conceptual reconstruction; not a source-figure transcription:
Choose oracle, vote, ranker, or test
Wrong selector choice turns extra compute into confident error. An analysis oracle gives an exact correctness judgment, such as evaluating 37 × 19 against 703; it is an upper-bound diagnostic, not assumed available at deployment. Voting chooses the most common answer. A learned verifier predicts correctness and must be calibrated. A ranker orders candidates; ranking is not verification. Deterministic tests run fixed assertions; they are incomplete specifications, strong only for covered behavior.
Protected tests and labels must be unavailable to generation and ranking. Only the deterministic test runner sees them after candidate generation; otherwise the system can copy answers, overfit fixtures, or turn calibration into leakage.
| Work | Best available signal | Known blind spot |
|---|---|---|
| Arithmetic | Exact calculation | Badly parsed question |
| Python repair | Public plus hidden tests | Missing behavioral case |
| SQL transformation | Expected rows, types, totals | Semantically wrong but invariant-preserving query |
| Support reply | Policy rules plus human rubric | Empathy and truth may remain subjective |
For SQL, suppose candidates transform orders(customer_id, cents) into customer totals. Verify schema, row count, SUM(cents) conservation, and three hidden fixtures containing refunds and duplicate customer IDs. A query can pass simple totals while assigning money to the wrong customer; add per-customer expected rows rather than trusting one aggregate invariant.
Test the rounding boundary nobody saw
ShipRight’s public test covers a comfortable case:
assert discounted_cents(1_000, 1_000) == 900
All four candidates pass. A hidden test covers the policy boundary:
assert discounted_cents(3, 5_000) == 1
Candidate A uses round(3 * 0.5) and returns 2 in one implementation. Candidate B uses integer floor division and returns 1. Candidate C hard-codes the public fixture. Candidate D raises on valid input. Public plus hidden results:
| Candidate | Public | Hidden | Eligible |
|---|---|---|---|
| A | pass | fail | no |
| B | pass | pass | yes |
| C | pass | fail | no |
| D | fail | not run | no |
The selector may recommend B for review. It must not silently patch production. Day 07 will make that boundary executable.
Distinguish selection strategies with evidence
Oracle coverage and selection accuracy are different. Oracle success asks whether any correct candidate exists. Selection accuracy asks whether the method chose a correct one. If candidate answers are [1, 2, 2, 2] and the exact answer is 1, oracle success is true while majority vote selects 2. More samples can make a correlated misconception more popular.
A learned ranker may prefer polished prose over correct arithmetic. Deterministic tests may overfit public cases. Counter each with held-out evidence: exact answers unknown to the generator, hidden fixtures, alternate orderings for rankers, and human review for policy-sensitive support.
Calibration records a confusion matrix: true positives, false positives, true negatives, and false negatives. It also records false-positive rate (FPR), false-negative rate (FNR), abstentions, test provenance, and behavior not covered by tests. Compiler or kernel optimization adds another limit: two programs may be equivalent on fixtures yet differ in undefined behavior or performance. Require semantic checks plus benchmark confidence intervals; abstain when evidence is insufficient. Open-ended support replies should usually abstain unless policy facts and human-quality review are available.
Normal path: B passes protected suites and is recommended for review. Failure path: all candidates fail or selector FPR exceeds threshold, so selection is none. Recovery: improve coverage or recalibrate on a disjoint set; never weaken assertions to create a winner. Cleanup: discard candidate workspaces after storing hashes and exits.
{
"schema": "selector-calibration/v1",
"confusion_matrix": {"tp": 18, "fp": 1, "tn": 27, "fn": 4},
"fpr": 0.0357,
"fnr": 0.1818,
"abstentions": 6,
"protected_test_provenance": "ci-suite@sha256:example",
"uncovered_behavior": ["currency overflow", "concurrent price updates"]
}
Scored check, with one unknown: candidates [1, 2, 2, 2], exact oracle 1. Which selector is valid: majority vote or exact oracle? Expected: exact oracle. Choosing vote confuses agreement with correctness.
Assessment receipt: given one unknown—the correct selector for [1, 2, 2, 2] when exact evaluation says 1—the accepted answer is exact oracle; evidence is direct equality, misconception is treating agreement as correctness, and remediation is comparing vote output with the supplied oracle. The confusion matrix remains calibration evidence, not a second scored item.
Day 03 hands selector-calibration.json to Day 04 beside difficulty-profile.json. Rule remains: no adequate evidence means abstain.