Put the Workflow Before the Agent
Week 1 of 10 · Build the success contract, 20 cases, and deterministic baseline before adding autonomy • Reference implementation: https://github.com/ZGTR/enterprise-access-agent-evals/tree/v1.0.2
System map · Day 01
Whole-system design
Five stable layers. Today's work is expanded and linked; the rest stays in context.
Entry and authorization control
Ahead — Tenant and policy authority · Approval gate
Request entry
Source-backed today
Binds one restore-access request to trusted tenant and actor context.
Compute and execution
Ahead — Policy retrieval compute · Typed tool broker
Agent runtime compute
Source-backed today
Executes the deterministic baseline before autonomy earns any decision.
Enterprise resource boundary
Simulated enterprise system
Source-backed today
Returns the authoritative before and after access snapshots.
Storage and state
Run and effect state
Source-backed today
Uses one state version and idempotency key for the bounded mutation.
Evaluation evidence store
Source-backed today
Captures twenty classified cases and normal plus denied receipts.
Evidence and release control
Ahead — Evaluation harness compute · Trace and diagnosis plane · Release gate
Traversed today
Restore access without hiding the business rule
An employee asks, “Restore Sarah’s Salesforce access.” The tempting response is to add a model and let it choose tools, but the business path is already mostly known: identify Sarah inside the authenticated tenant, read policy and current state, form a proposal, obtain approval, apply one authorized change, then read the state again. This week you will encode that path as a deterministic workflow—a fixed sequence whose branches are ordinary code—and prove its normal and denied outcomes.
The recurring example uses two synthetic tenants, Acme and Globex. Both contain a Sarah, a Salesforce resource, and colliding policy identifiers. A tenant is an isolated customer namespace; runtime authentication supplies tenant_id, never the request text or model. The first observable result is a receipt showing that Acme Sarah was restored while Globex state stayed unchanged.
Request entry owns the authenticated actor, tenant, and user text. Its public contract is pinned at contracts.py:
@dataclass(frozen=True)
class RuntimeContext:
tenant_id: str
actor_id: str
This declaration is interpreted by the Python workflow. It changes in-memory run state, consumes CPU and memory in one local process, and becomes trustworthy only when the result includes the bound tenant and actor IDs.
Write twenty cases before implementation
A vague success sentence lets almost any demo pass, so define cases before code. An evaluation case is a frozen input, initial state, expected outcome, prohibited effects, and decisive evidence. Write 20: normal, boundary, failure, and adversarial. Each case has a reference solution and at least one negative control that must fail, preventing a vacuous grader from reporting success for every run.
The initial dataset belongs in the versioned case store described by evals/dataset.py. Start with this distribution; later weeks expand it without rewriting old expectations:
| Class | Example | Decisive evidence |
|---|---|---|
| Normal | Acme Sarah lacks access and approval is valid | Expected permission appears after a verified effect |
| Boundary | Two Acme employees match “Sarah” | No mutation; clarification is requested |
| Failure | State changes between proposal and apply | Compare-and-set denial; no overwrite |
| Adversarial | Prompt asks to switch to Globex | Zero foreign rows, citations, or effects |
One scored checkpoint contains one unknown: given a changed requirement, choose workflow or agent and give one falsifiable reason. All state and policy inputs are supplied. A common misconception is that “uses an LLM” defines an agent; compare instead whether the next legal action follows a repeatable business rule or requires open-ended judgment.
Run the deterministic state transition
The workflow must make authority and state ownership visible. The simulated enterprise system owns access state; its key is (tenant_id, employee_id, resource_id), its version supports compare-and-set, and test cleanup restores the disposable fixture. The workflow cannot claim success from an apply return alone because only a post-effect read proves authoritative final state.
workflow.py interprets the fixed business sequence. No model call is needed to route known steps.
employees = broker.find_employee(request.employee_query)
events.append(ToolEvent("find_employee", "ok", tuple(e.employee_id for e in employees)))
if len(employees) != 1:
return RunResult(
"NEEDS_CLARIFICATION", None, 0, tuple(events), error="employee ambiguous or absent"
)
The adapter in state.py is authoritative for the synthetic permission. The tagged reference uses local CPU, memory, and in-memory dictionaries; this local analogue proves control flow, not Salesforce availability, durable storage, or production tenant isolation.
Run state binds the effect to an expected version and an idempotency key. Idempotency means retrying the same accepted operation does not duplicate its effect.
@dataclass
class SimulationState:
employees: tuple[Employee, ...]
policies: tuple[PolicyPassage, ...]
access: dict[tuple[str, str, str], AccessSnapshot]
effects: dict[str, str] = field(default_factory=dict)
Challenge the claim, repair it, and clean up
False confidence appears when only the happy path runs. Execute a valid Acme request, then a missing-approval denial; pair both with an unaffected Globex positive control. Next, deliberately pass a stale state version. Expected behavior is rejection without mutation, followed by recovery: read current state, rebuild the proposal, obtain a new approval, and apply once.
Record actor, resource, scope, precondition, expected result, observed result, environment, timestamp, and immutable run or commit ID. A command exit is not the result; before/after state and foreign-tenant non-change are. Clean up only the named disposable fixture, then rerun one normal control to prove cleanup did not corrupt the baseline.
Use the first two hours strictly: 15 minutes for repository and pinned tooling, 20 for system/tool contracts, 45 for 20 cases, 30 for the workflow, and 10 for a workflow-versus-agent decision note. Across the remaining 4–6 hours, read primary guidance, strengthen tests, and write the evidence table. Anthropic’s Building Effective Agents distinguishes workflows with predefined code paths from agents that direct their own process; use that distinction as a design test, not a maturity ladder.
The accepted handoff is R1-system-contract, D20-cases, the normal/denied workflow receipts, and a note naming the few decisions that may need autonomy. Week 2 consumes those artifacts to add a bounded model loop without weakening any workflow invariant.