Compose a Fixed Inference Architecture
Paper version: “Archon: An Architecture Search Framework for Inference-Time Techniques,” v6 — https://arxiv.org/abs/2409.15254v6 • Reference implementation at inspected commit: https://github.com/ScalingIntelligence/Archon/commit/07114d77af283b6e8185a49ebf22216fdbbf2a55 • Source: Stanford CS329A, “Test-Time Compute Scaling,” 00:43:20–01:01:30 — https://www.youtube.com/watch?v=-Ggc37xLj_Y&t=2600s
Turn an allocation policy into one reproducible pipeline
Day 04 produced allocation-policy.json. Today’s only new capability is composing one fixed inference architecture: a graph of modules with explicit inputs, outputs, and call budgets. You will produce fixed-architecture.yaml; comparison and search wait until Day 06.
Archon is a published framework for combining inference-time techniques and models (paper v6, official repository at inspected commit). This ShipRight pipeline is a conceptual reconstruction, not Archon’s schema, source-figure transcription, or reported experiment.
Give each module one non-overlapping job
A generator proposes candidates. A revision loop edits one candidate using specific feedback. A critic identifies concrete defects but does not edit. A learned verifier estimates correctness and may abstain. A ranker orders candidates; it cannot convert a prediction into proof. A fuser combines useful parts into a new candidate. A deterministic test runner executes finite checks. selected-answer is a recommendation backed by evidence, never an automatic release.
architecture_id: shipright-fixed-v1
input: allocation-policy.json
budget:
max_calls: 8
max_wall_seconds: 30
modules:
- {id: generator, samples: 3, calls: 3, output: candidate-store}
- {id: critic, inputs: [candidate-store], calls: 1}
- {id: revision-loop, inputs: [candidate-store, critic], calls: 1, max_steps: 1}
- {id: learned-verifier, inputs: [candidate-store], calls: 1, may_abstain: true}
- {id: ranker, inputs: [critic, learned-verifier], calls: 1, keep: 2}
- {id: fuser, inputs: [ranker], calls: 1, outputs: 1}
- {id: deterministic-test-runner, inputs: [candidate-store, fuser], protected: true}
selection: smallest_passing_artifact_then_lowest_index
output: evidence-store/fixed-run-receipt.json
Protected tests are not an input to generator, critic, learned verifier, ranker, or fuser. Test results can authorize a review recommendation only after generation stops. This boundary prevents a sophisticated module graph from becoming a fixture-copying machine.
max_calls counts model calls: three generator samples plus one call each for critic, revision, learned verifier, ranker, and fuser equals eight. Deterministic test executions are recorded separately as test_runs; they do not disappear inside the model-call budget.
Trace the ShipRight repair through fixed roles
The pricing bug enters as task. budget-router reads the Day 04 policy and grants three parallel candidates plus one revision. generator writes three hashes to candidate-store. critic flags floating-point rounding. learned-verifier scores likely policy compliance but abstains on negative-price behavior. ranker retains two candidates. fuser produces one new integer-arithmetic candidate. deterministic-test-runner evaluates source and fused candidates against public plus protected suites. Only passing hashes reach selected-answer.
Normal path: one source candidate passes; fuser failure cannot erase it. Failure: critic and learned verifier agree on a candidate that protected tests reject. Recovery: selected-answer=NO_SELECTION; do not override tests with model consensus. Positive control: unrelated shipping-fee fixture passes in every copied workspace. Cleanup: candidate bodies expire after evidence-store records hashes and exits.
For customer support, learned-verifier and ranker may enforce policy phrases, but fuser cannot prove empathy or truth when account context is absent. Architecture must abstain. For a compiler kernel, deterministic equivalence tests remain incomplete; require numerical tolerances, undefined-behavior checks, and measured performance before recommending review.
Freeze the architecture before comparing it
Changing modules while reading results makes evaluation uninterpretable. Validate YAML, pin model/config/test fingerprints, and run one smoke fixture. Record calls, input tokens, output tokens, wall time, test runs, and USD as separate fields; do not collapse them into “compute.”
Assessment receipt: given one unknown—fused candidate fails protected tests while a source candidate passes—the accepted selection is the passing source candidate; evidence is test status, misconception is assuming fusion dominates inputs, and remediation is replaying both hashes against the same suite.
Day 05 hands immutable fixed-architecture.yaml plus fixed-run-receipt.json to Day 06. Day 06 may vary bounded config fields on a development split, never on untouched test tasks.