Trace the First Causal Failure
Week 4 of 10 · Trace every causal boundary and diagnose 20 runs from first divergence • Public source target: https://github.com/ZGTR/enterprise-access-agent-evals/tree/v1.0.2
System map · Day 04
Whole-system design
Five stable layers. Today's work is expanded and linked; the rest stays in context.
Entry and authorization control
Covered — Request entry · Tenant and policy authority · Approval gate
Compute and execution
Agent runtime compute
Design target · not proved
Emits correlated model and decision spans for one complete run.
Policy retrieval compute
Design target · not proved
Records retrieval inputs and citation IDs without policy text leakage.
Typed tool broker
Design target · not proved
Records validated arguments, denials, results, latency, and effect IDs.
Enterprise resource boundary
Covered — Simulated enterprise system
Storage and state
Covered — Run and effect state
Evaluation evidence store
Design target · not proved
Keeps a redacted twenty-run diagnosis table and instrumentation defects.
Evidence and release control
Ahead — Evaluation harness compute · Release gate
Trace and diagnosis plane
Source-backed today
Finds the earliest causal failure across eleven mutually exclusive labels.
Traversed today
Diagnose causes, not disappointing answers
Week 3 produced R3-tenant-proof, 40 cases, and a tenant-scoped policy corpus. A failed final answer still does not say whether the first error was misunderstood intent, bad retrieval, wrong tool arguments, rejected execution, or incorrect state interpretation. This week you add correlated traces—ordered records of operations inside one run—and label the earliest event that made success impossible.
A span records one operation with start/end time, status, bounded attributes, and parent relationship. Trace the model decision, retrieval, tool validation and call, approval check, state read/write, final verification, latency, tokens, and estimated cost. Do not record raw prompts, secrets, policy bodies, or personal data.
Tagged agent.py does not yet wire the trace recorder through each decision. Adding one run ID and preserving parent/child order is today’s design target so a late communication failure remains distinguishable from an early wrong employee choice.
Instrument retrieval and tools at their owners
One giant “agent.run” span hides the boundary that failed. Instrument work where authority and state change: retrieval owns candidate evidence; broker owns validation and tool calls; adapter owns authoritative effects.
The tagged retrieval.py proves authorization-before-ranking; retrieval-span enrichment remains a design target:
scoped = (policy for policy in policies if policy.tenant_id == tenant_id)
terms = {term.casefold() for term in query.split()}
tools.py supplies typed authorization errors and effect receipts. Adding tool name, schema version, argument digest, authorization result, effect ID, and latency to allowlisted spans is today’s design target. An argument digest proves which validated input ran without leaking its contents.
The OpenTelemetry adapter at telemetry.py exports redacted spans. OpenTelemetry’s semantic conventions define common telemetry names; GenAI conventions are evolving, so pin the package/convention version and keep application-specific attributes bounded.
def record(self, kind: str, status: str, attributes: Mapping[str, object]) -> TraceSpan:
safe = {key: value for key, value in attributes.items() if key in ALLOWED_ATTRIBUTES}
span = TraceSpan(self.run_id, len(self.spans) + 1, kind, status, safe, monotonic_ns())
self.spans.append(span)
return span
Assign one earliest primary cause
Multiple symptoms can follow one mistake. If retrieval selects a stale policy, the model proposes the wrong permission and the final message becomes false; retrieval is the earliest causal failure. Record exactly one primary label plus supporting span IDs:
- intent misunderstanding; 2. planning; 3. retrieval; 4. tool selection; 5. tool arguments; 6. execution; 7. state interpretation; 8. recovery; 9. authorization or safety; 10. final communication; 11. evaluation defect.
An evaluation defect means the system succeeded but the case, reference, instrumentation, or grader was wrong. It prevents teams from “fixing” a healthy agent to satisfy a broken test.
The case loader in evals/dataset.py supplies frozen inputs. Produce a separate 20-run diagnosis table with run ID, outcome, earliest label, decisive span, component, and reviewer note; that table is an exercise artifact, not present in the tag. Add broken tracing, missing parent, truncated tool result, stale state interpretation, and evaluator-defect controls during the staged 45-case milestone.
Break one span, repair evidence, and clean up
Run a normal access restore and a missing-approval denial, with Globex unchanged as positive control. Then drop the retrieval span. The result may be correct, but causal diagnosis must return NO_VERDICT where decisive evidence is absent—not guess from the final message.
Recovery restores instrumentation, replays the same frozen case under a new run ID, and checks parent order plus state outcome. Cleanup deletes only disposable trace exports; case and run digests remain. Keep a local analogue caveat: one process can prove correlation logic but not distributed collector delivery, retention, or production privacy.
The scored checkpoint supplies one complete trace; the only unknown is earliest causal label. Feedback contrasts the worst visible symptom with the first wrong boundary. Decline multiple primary causes. Spend 1.5 hours with the Phoenix tracing tutorial and Hugging Face observability unit, 4–5 hours instrumenting and analyzing, and 1 hour writing the failure ledger.
The handoff is R4-trace-schema, D45-cases, and a 20-run failure ledger. Week 5 consumes these receipts to build a portable dataset → solver → scorer harness.