Search on Development, Evaluate Once
Paper version used: Archon v6 — https://arxiv.org/abs/2409.15254v6 • Reference implementation at inspected commit: https://github.com/ScalingIntelligence/Archon/commit/07114d77af283b6e8185a49ebf22216fdbbf2a55 • Lecture account: Stanford CS329A, 00:43:20–01:01:30 — https://www.youtube.com/watch?v=-Ggc37xLj_Y&t=2600s
Compare bounded configurations without touching final test
Day 05 froze fixed-architecture.yaml and its smoke receipt. Today’s one new capability is searching a bounded set of architecture configurations on a development split, selecting by explicit quality–cost rules, then evaluating the winner once on untouched test tasks. Output is architecture-selection.json.
Archon v6 says its automated search uses a 20% development set and supports restrictions including total inference calls, layers, models, and budget (paper v6). That is source-reported methodology. ShipRight’s smaller search below is a conceptual reconstruction, not a reproduction or source-figure transcription. Lecture and paper versions can report different summary metrics; never combine them into one claim.
Declare search space and splits before results
Searching an unlimited graph spends uncontrollable compute and overfits noise. Freeze task IDs into three disjoint partitions: development for configuration comparison, validation for tie-breaking and early stop, and untouched test for one final estimate. No test task, label, trace, or protected fixture may influence architecture choice.
{
"search_id": "shipright-arch-search-001",
"development_task_ids": ["price-001", "price-004", "price-009", "price-012"],
"validation_task_ids": ["price-014", "price-018"],
"untouched_test_task_ids": ["sealed-by-ci"],
"configurations": [
{"id": "width-2", "samples": 2, "revision_steps": 0, "critic": false, "fuser": false},
{"id": "width-3", "samples": 3, "revision_steps": 0, "critic": false, "fuser": false},
{"id": "width-2-revise-1", "samples": 2, "revision_steps": 1, "critic": false, "fuser": false},
{"id": "critic-rank-fuse", "samples": 3, "revision_steps": 0, "critic": true, "fuser": true}
],
"limits": {"max_configs": 4, "max_calls_per_task": 8, "max_wall_seconds_per_task": 30}
}
The list is the complete search order, so max_configs=4 is deterministic. Reject a listed configuration that exceeds per-task limits before execution. Keep generator, deterministic test runner, selector rule, and protected-test boundary fixed so comparison has a defined meaning.
Measure cost dimensions separately
One “compute” number hides trade-offs. Record model calls, input tokens, output tokens, wall time, deterministic test runs, and USD separately for every task/config pair. Tokens measure model work; wall time includes parallelism and queues; test runs consume local compute; USD depends on provider prices and cache rules.
The development receipts below are synthetic, deterministic teaching data; they are not Archon results or evidence that one architecture improves a live model.
| Config | Dev passes | Validation passes | Calls | Input tokens | Output tokens | Wall seconds | Test runs | USD |
|---|---|---|---|---|---|---|---|---|
| width-2 | 3/4 | 1/2 | 8 | 8,000 | 2,400 | 4.0 | 8 | 0.08 |
| width-3 | 4/4 | 2/2 | 12 | 12,000 | 3,600 | 4.5 | 12 | 0.12 |
| width-2-revise-1 | 4/4 | 2/2 | 12 | 15,000 | 4,200 | 7.0 | 12 | 0.15 |
| critic-rank-fuse | 4/4 | 2/2 | 24 | 30,000 | 8,000 | 10.0 | 12 | 0.35 |
In real work, every value must come from a signed run receipt. For this worked fixture, select highest development passes, then highest validation passes, then lowest USD, calls, and wall time. This runnable scorer derives the documented winner instead of asserting it by hand:
rows = [
{"id": "width-2", "dev": 3, "validation": 1, "usd": 0.08, "calls": 8, "wall": 4.0},
{"id": "width-3", "dev": 4, "validation": 2, "usd": 0.12, "calls": 12, "wall": 4.5},
{"id": "width-2-revise-1", "dev": 4, "validation": 2, "usd": 0.15, "calls": 12, "wall": 7.0},
{"id": "critic-rank-fuse", "dev": 4, "validation": 2, "usd": 0.35, "calls": 24, "wall": 10.0},
]
winner = min(rows, key=lambda row: (-row["dev"], -row["validation"], row["usd"], row["calls"], row["wall"]))
assert winner["id"] == "width-3"
Also inspect the Pareto frontier: another configuration dominates only when it is no worse on every declared objective and better on at least one. Lock width-3 before opening test.
Open untouched test once and preserve failures
Normal path: four bounded configs finish development; three tie on development and validation passes; the predeclared cost tie-break selects width-3, which receives one untouched-test run. Failure: winner’s test pass rate falls or selector false positives appear. Do not reopen search using test feedback. Recovery is a new versioned experiment with a new held-out split, not mutation of this receipt. Positive control: baseline single-sample architecture runs on identical tasks. Cleanup removes candidate bodies while retaining task/config fingerprints, metric vectors, exits, and selected config.
{
"schema": "architecture-selection/v1",
"winner": "width-3",
"run_profile": {"samples": 3, "workers": 3, "budget_units": 3},
"selection_split": "development_plus_validation",
"test_opened_once": true,
"metrics": ["calls", "input_tokens", "output_tokens", "wall_seconds", "test_runs", "usd"],
"release_decision": "eligible_for_local_capstone"
}
Assessment receipt: given one unknown—config B matches A’s validation passes while using fewer calls and no more of any other metric—the accepted answer is “A is dominated”; evidence is metric vector, misconception is optimizing accuracy alone, and remediation is checking every declared objective.
Day 06 hands architecture-selection.json and untouched-test receipt to Day 07. Day 07 implements only compute allocation and candidate selection; it does not recreate architecture search.