Operate and Support with Observable Evidence
Diagnose from customer symptom to substrate while treating every green signal as evidence for only its own boundary.
System map · Day 14
Whole-system design
Five stable layers. Today's work is expanded and linked; the rest stays in context.
Product and authority
Covered — People and product entry points · Identity and policy · HelixWorks control plane · Generated application plane
Delivery and desired state
Covered — Git desired state · CI and immutable artifacts · Argo CD reconciliation
Cloud and orchestration
Covered — Terraform and AWS APIs · Accounts, VPC, DNS, and private paths
Kubernetes or EKS control plane
Source-backed today
Separates API and controller health from the health of workloads and the customer request path.
Compute and traffic
Covered — Worker compute · Generated app workloadsAhead — Ambient mesh data plane
Platform service workloads
Source-backed today
Exposes health and service evidence for each independently progressing provider responsibility.
Storage and evidence
Ahead — Infrastructure state · Cluster desired and live state
Product data and artifacts
Source-backed today
Keeps tenant-filtered durable evidence queryable when transient process logs disappear.
Evidence and observability
Source-backed today
Correlates audit, logs, metrics, traces, controller state, and customer probes into a diagnosis path.
The enterprise problem and today’s slice
Enterprise problem: A HelixWorks application can show green deployment and GitOps status while customers receive slow, denied, or incorrect responses, so support teams need evidence that crosses boundaries without exposing tenant data. Whole-course context: The incoming evidence is a release with known artifact and local rollback records; today defines the larger operating evidence required around it. Today’s slice: We inspect the implemented tenant-scoped evidence store and distinguish that local proof from future traces, metrics, Kubernetes, Argo CD, cluster-API, and AWS observations. End-of-day evidence: The runnable proof is a durable tenant-filtered event plus local health and lifecycle results; the complete support bundle remains the production target. Still unsolved: Customer export, application retirement, durable deletion, and live AWS operational proof remain deferred.
Customer outcome and implementation focus
The customer outcome is a reviewable operate and support with observable evidence change, not a collection of requirements. This day starts with the implementation boundary, then uses the command or manifest below to produce positive, denied, and recovery evidence.
Components in focus
Telemetry collector, metrics/log/trace backends, support API, and evidence service; workload and collector pods; metrics/log stores, trace store, and immutable incident evidence; cache: not involved.
This map names the implementation boundary for this day. The service or controller changes only the state it owns; runtime and audit evidence let the operator distinguish a declared change from an effective one.
Operate from evidence rather than controller colour
Diagnose boundaries in dependency order
The same symptom can originate in hardware, cluster API, workload, delivery, or application policy, so read each boundary without mutating it first. The current repository has no live AWS or end-to-end telemetry fixture; its concrete evidence boundary is the tenant-filtered SQL event store. Read the exact implementation in services/shared/persistence.py.
def events(self, organization_id: str) -> list[dict[str, object]]:
return [json.loads(row[0]) for row in self.database.all("SELECT payload FROM evidence_events WHERE organization_id=? ORDER BY observed_at, evidence_id", (organization_id,))]
Key takeaways
Operational truth is a chain of scoped observations, not one universal green badge.
- Start with customer evidence, then diagnose substrate to service without skipping owners.
- GitOps health, Kubernetes readiness, and AWS existence prove different boundaries.
- Redacted correlation IDs connect evidence without copying customer domain data.
Checklist
A support case is actionable only when evidence names its source, freshness, and owner.
- [ ] Customer symptom and unaffected control are reproducible
- [ ] Every observation is read-only until the failing owner is identified
- [ ] Telemetry schema forbids secrets and raw domain records
- [ ] Recovery contains environment, run, release, trace, and timestamp evidence