Measure Retrieval Before Generation

Week 6 left calibrated independent graders and a 75-case suite. A wrong policy answer can still originate in two different systems: retrieval may omit the decisive passage, or generation may ignore a passage that was present. This week you record retrieved IDs separately from generated claims and effects, then reach 85 cases with irrelevant, stale, conflicting, injected, and foreign-tenant documents.

Recall@k asks whether the required passage appears among the top k allowed results. Context precision asks how much retrieved context was relevant. Groundedness checks whether claims follow supplied evidence; citation correctness checks whether each citation supports its claim. Tenant-leak rate is a hard zero-tolerance invariant.

The policy authority in contracts.py pins tenant and as-of version. Stale/conflicting same-tenant documents are test inputs; foreign documents remain unauthorized.

Create controlled corpus variants: one irrelevant near-duplicate, one stale superseded rule, two conflicting current passages, one indirect prompt injection, and one colliding Globex policy. Changing one variable at a time preserves causal diagnosis.

Tagged retrieval.py returns tenant-scoped passages with citation IDs. Persisting scores and corpus digest beside those IDs is this week’s extension:

def scoped_policy_search(
    policies: tuple[PolicyPassage, ...], tenant_id: str, query: str, limit: int = 5
) -> tuple[PolicyPassage, ...]:
    """Authorize namespace before ranking; retain stale/conflicting versions for diagnosis."""
    scoped = (policy for policy in policies if policy.tenant_id == tenant_id)

If required evidence is absent, classify retrieval. If it is present and the answer contradicts it, classify generation or state interpretation. Never tune retriever and prompt in the same experiment.

evals/task.py computes metrics from frozen expected IDs and recorded outputs. Ragas offers metric implementations and concepts in its catalogue, but external scores remain supplementary to executable tenant and state assertions.

evals/dataset.py reaches 85 cases and stores corpus digest, retrieved IDs, generated citations, effect IDs, metric denominators, and no-answer expectation. Report each metric separately by difficulty and risk.

A useful RAG experiment must show which subsystem changed and preserve a legal control. Run a current Acme policy restore and an injected instruction asking for Globex data; pair them with an unaffected positive control from the same tenant. Then force the correct passage below k. The expected failure is retrieval even if the model gives a plausible answer. Recovery adjusts one retrieval parameter or corpus rule, reruns the frozen case, then verifies citation support and zero leakage. Cleanup removes only disposable corpus variants and rebuilds the original index.

The scored checkpoint supplies retrieved IDs and final claims; the unknown is retrieval or generation failure. Feedback contrasts a grounded fluent answer with complete retrieval. Decline simultaneous prompt and retriever changes. Spend 1.5 hours on the RAGAS paper and metric docs, 4–5 hours on corpus injections and graders, and 1 hour writing a boundary-specific report.

The handoff is R7-rag-report, D85-cases, and corpus v2. Week 8 repeats every case five times to distinguish occasional capability from consistent reliability.