Ship, Observe, and Govern

Ordinary tests can prove schemas, routing functions, storage adapters, and idempotency behavior. They cannot alone prove that a new prompt, model, retriever, or memory corpus preserves behavior. Production CI/CD needs both deterministic tests and versioned behavioral evaluations.

Store evaluation datasets in version control or a versioned registry. Pin model configuration and record prompt, procedure, embedding, and corpus versions so a result can be reproduced.

Different subsystems need different measures. A single “agent score” hides dangerous trade-offs.

An unsafe-action regression should block release even if average task success improves.

The pipeline runs deterministic and behavioral gates, builds an immutable image, signs it, deploys to staging, and promotes gradually. CI should authenticate to the cloud with OIDC or workload identity rather than stored long-lived keys.

name: agent-ci
on: [pull_request]
permissions:
  contents: read
  id-token: write

jobs:
  verify:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: {python-version: "3.12"}
      - run: pip install -r requirements.lock
      - run: pytest -q
      - run: python evals/run.py --suite regression
      - run: python evals/run.py --suite memory-poisoning
      - run: terraform fmt -check -recursive
      - run: terraform validate

Production promotion should reference an image digest and evaluation run, not rebuild mutable source. Canary thresholds should support automatic halt and an operator-driven rollback to the previous known-good bundle.

Terraform owns networks, IAM, PostgreSQL, caches, queues, object storage, secrets references, backup policy, observability sinks, and deletion protection. Helm—or another deployment layer—owns Kubernetes workloads and their runtime configuration.

module "postgres" {
  source            = "./modules/postgres"
  high_availability = true
  backups_days      = 14
  deletion_protection = true
}

module "memory_queue" {
  source            = "./modules/queue"
  dead_letter_queue = true
}

module "agent_service" {
  source   = "./modules/kubernetes-service"
  image    = var.agent_image_digest
  replicas = 3
}

Use separate state and approval policy per environment. Review plans before production applies. Workloads receive narrowly scoped identities; secrets are referenced from a managed secret store and never embedded in Terraform state or Helm values.

OpenTelemetry-compatible traces should connect webhook ingress, checkpoint load, procedure resolution, episodic retrieval, triage, semantic search, model calls, tool authorization, provider calls, response, queue publication, and background memory writes.

with tracer.start_as_current_span("memory.search") as span:
    span.set_attribute("tenant_hash", tenant_hash)
    span.set_attribute("memory_type", "semantic")
    span.set_attribute("top_k", 5)
    results = search_memory_safely()
    span.set_attribute("result_count", len(results))

memory_search_latency.observe(elapsed)
tool_calls_total.labels(
    tool="schedule_meeting",
    outcome="authorized_success",
).inc()

Use hashed or low-cardinality identifiers in metrics. Raw email content, prompts containing personal data, access tokens, and secrets do not belong in telemetry. Keep an access-controlled audit trail for decisions that require reconstruction.

Track task success, unsafe-action rate, memory hit rate, stale-memory rate, correction rate, queue age, provider errors, checkpoint latency, token use, and cost. An alert should connect to a runbook and an owner, not merely create another dashboard line.

Every request carries authenticated tenant_id, user_id, thread_id, and an idempotency key. Database policy and queries enforce tenant scope. Tool implementations re-authorize every side effect. Memory records support access, export, correction, expiry, supersession, quarantine, and deletion.

Procedural writes require authenticated authority and approval. Retrieved memory cannot override system safety rules. Sensitive candidates are rejected or routed to controlled review. Encryption protects data in transit and at rest, while application-level redaction limits what is stored in the first place.

Backup and recovery deserve executable tests. Define recovery-point and recovery-time objectives, automate database and object-store backups, maintain vector snapshots when a dedicated index exists, and run restore drills. A cache is rebuilt; the authoritative database is restored.

Alice’s provider webhook is authenticated and deduplicated by provider message ID. The service loads the graph checkpoint using tenant, user, and thread identity. It resolves approved triage procedures and retrieves reviewed episodes. Structured triage returns respond.

The response agent searches active semantic memories and retrieves “no meetings before 10:00.” It calls the calendar read tool, which checks scope and returns free slots. The agent proposes 14:00; because the procedure requires approval for external attendees, it drafts a reply instead of scheduling immediately. Tool arguments and the procedure version are recorded in the trace.

The response commits, and an outbox publishes a learning event. A background worker loads redacted source content, sees no new durable fact, records the successful episode after feedback, and advances no procedure. Online metrics feed the next evaluation dataset. Every transition has a durable identifier and a responsible authority.

Before launch, prove rather than assume: cross-tenant retrieval returns zero records; replay creates no duplicate effect; an injected email cannot write procedure; a superseded fact disappears from retrieval; a model timeout degrades safely; a queue retry is idempotent; a database restore works; a candidate rollout can return to the previous prompt, model, and index bundle.

This is the final mental model: the LLM supplies reasoning and language, LangGraph supplies controlled execution, tools supply authorized capabilities, and the memory system supplies governed continuity. Production engineering makes each of those claims observable and reversible.

• Treat prompts, models, retrieval, and memory policy as versioned behavioral release artifacts.
• Combine deterministic tests with triage, retrieval, tool, memory, security, and end-to-end evaluations.
• Terraform and Helm make infrastructure, identity, backup, and runtime configuration reproducible.
• OpenTelemetry traces must span both the hot path and the background learning lifecycle without leaking content.
• Production readiness means safe authority, observable behavior, tested recovery, and fast rollback.

• [ ] I can define a CI/CD gate for each agent subsystem.
• [ ] I can explain how OIDC avoids long-lived CI cloud credentials.
• [ ] I can assign infrastructure ownership between Terraform and Helm.
• [ ] I can trace one email from ingress through learning-event processing.
• [ ] I can demonstrate tenant isolation, idempotency, poisoning defense, restore, and rollback before launch.