Service Traffic Baseline: Why Add a Mesh?

Enterprise problem: A service mesh can appear to improve security while silently breaking a customer publish request, because the team never recorded which service names, ports, workload identities, and NetworkPolicies already made that request work. Whole-course context: The incoming HelixWorks application is rendered from Git into a local Kubernetes cluster; today captures its pre-mesh traffic contract before any Istio component or namespace label is added. Today’s slice: We inspect the five Services, five Deployments, five ServiceAccounts, and four effective NetworkPolicies, then run the checked-in kindnet positive/negative packet probe before explaining the remaining identity, encryption, and application-aware policy gap. End-of-day evidence: Only a successful probe may seal a source-revision-bound baseline containing reviewed Pod names and UIDs, kindnet status, exact policies, two allowed controls before and after, and one exact denied timeout. Still unsolved: Installing Istio, enrolling the HelixWorks namespace (helixworks-forge), mutual TLS, waypoints, mesh authorization, telemetry, failure recovery, and rollback remain deferred.

The customer outcome is a reviewable service traffic baseline: why add a mesh? change, not a collection of requirements. This day starts with the implementation boundary, then uses the command or manifest below to produce positive, denied, and recovery evidence.

Kubernetes Services, CoreDNS, kube-proxy/CNI, application pods, and telemetry collector; node/pod networking; Kubernetes etcd plus telemetry stores; cache: not involved in the request baseline.

This map names the implementation boundary for this day. The service or controller changes only the state it owns; runtime and audit evidence let the operator distinguish a declared change from an effective one.

Read the source contract exactly

A mesh migration can fail before any mesh component is involved if a Service selector, port, or identity is already wrong. These excerpts are exact slices of gitops/apps/forge/base/services.yaml, workloads.yaml, and network-policy.yaml in the implementation repository.

apiVersion: v1
kind: Service
metadata: { name: control-plane, namespace: helixworks-forge }
spec: { selector: { app.kubernetes.io/name: control-plane }, ports: [{ name: http, port: 8080, targetPort: http }] }

apiVersion: v1
kind: ServiceAccount
metadata: { name: control-plane, namespace: helixworks-forge }
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: control-plane, namespace: helixworks-forge }
spec:
  replicas: 0
  selector: { matchLabels: { app.kubernetes.io/name: control-plane } }
  template:
    metadata: { labels: { app.kubernetes.io/name: control-plane, app.kubernetes.io/part-of: helixworks-forge } }
    spec:
      serviceAccountName: control-plane

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny, namespace: helixworks-forge }
spec: { podSelector: {}, policyTypes: [Ingress, Egress] }
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: broker-publish-subscribe, namespace: helixworks-forge }
spec:
  podSelector: { matchLabels: { app.kubernetes.io/name: broker } }
  policyTypes: [Ingress]
  ingress:
    - from: [{ podSelector: { matchExpressions: [{ key: app.kubernetes.io/name, operator: In, values: [control-plane, evidence] }] } }]
      ports: [{ protocol: TCP, port: 8080 }]

Reading YAML cannot prove runtime behavior, and changing policy before collecting evidence destroys the comparison. Render and deploy the local overlay, review the exact control-plane, evidence, broker, and generator Pods, then run the repository’s checked-in positive/negative probe before any Ambient label exists.

git rev-parse HEAD
kubectl kustomize gitops/apps/forge/overlays/local >/tmp/forge-baseline.yaml
kubectl apply -f /tmp/forge-baseline.yaml
kubectl -n helixworks-forge rollout status deployment --all --timeout=180s
ALLOWED_SOURCE_POD="$(kubectl -n helixworks-forge get pods -l app.kubernetes.io/name=control-plane -o jsonpath='{.items[0].metadata.name}')"
ALLOWED_SOURCE_POD_UID="$(kubectl -n helixworks-forge get pod "$ALLOWED_SOURCE_POD" -o jsonpath='{.metadata.uid}')"
DENIED_SOURCE_POD="$(kubectl -n helixworks-forge get pods -l app.kubernetes.io/name=evidence -o jsonpath='{.items[0].metadata.name}')"
DENIED_SOURCE_POD_UID="$(kubectl -n helixworks-forge get pod "$DENIED_SOURCE_POD" -o jsonpath='{.metadata.uid}')"
DENIED_SOURCE_CONTROL_POD="$(kubectl -n helixworks-forge get pods -l app.kubernetes.io/name=broker -o jsonpath='{.items[0].metadata.name}')"
DENIED_SOURCE_CONTROL_POD_UID="$(kubectl -n helixworks-forge get pod "$DENIED_SOURCE_CONTROL_POD" -o jsonpath='{.metadata.uid}')"
TARGET_POD="$(kubectl -n helixworks-forge get pods -l app.kubernetes.io/name=generator -o jsonpath='{.items[0].metadata.name}')"
TARGET_POD_UID="$(kubectl -n helixworks-forge get pod "$TARGET_POD" -o jsonpath='{.metadata.uid}')"
NETWORK_POLICY_EVIDENCE_DIR="$(mktemp -d "${TMPDIR:-/tmp}/forge-network-policy.XXXXXX")"
export ALLOWED_SOURCE_POD ALLOWED_SOURCE_POD_UID DENIED_SOURCE_POD DENIED_SOURCE_POD_UID
export DENIED_SOURCE_CONTROL_POD DENIED_SOURCE_CONTROL_POD_UID TARGET_POD TARGET_POD_UID NETWORK_POLICY_EVIDENCE_DIR
printf 'review allowed=%s/%s denied=%s/%s control=%s/%s target=%s/%s\n' "$ALLOWED_SOURCE_POD" "$ALLOWED_SOURCE_POD_UID" "$DENIED_SOURCE_POD" "$DENIED_SOURCE_POD_UID" "$DENIED_SOURCE_CONTROL_POD" "$DENIED_SOURCE_CONTROL_POD_UID" "$TARGET_POD" "$TARGET_POD_UID"
NETWORK_POLICY_PROBE_APPROVED=1 make cni-probe

Approval is valid only after reviewing the printed names and immutable UIDs. make cni-probe invokes scripts/probe-network-policy.sh, which first runs scripts/verify-kind-network-policy.sh. That verifier requires the pinned Kind version, exact kind-$CLUSTER_NAME context, expected kindnet image, one ready kindnet agent per node, and ready nodes. The probe then vetoes non-local or already Ambient-labelled namespaces and refuses ambiguous, unready, wrongly owned, or changed Pods.

The experiment resolves the exact generator Pod IP. It proves control-plane-to-generator and evidence-to-broker reachability, attempts evidence-to-generator directly, and accepts only the script’s timeout status 42 as the denied result. Connection success, refusal, DNS failure, or any other status is inconclusive and fails the run. Both allowed controls run again afterward, and all four Pod identities are revalidated.

Do not claim a baseline merely because the command was printed or because four NetworkPolicy objects exist. A successful run creates new identities.txt, network-policies.yaml, positive-controls.txt, negative-stderr.txt, and kindnet-status.txt files in the pre-existing absolute evidence directory. Seal those artifacts with the source revision, context, time, and run ID. If the command does not complete successfully or any artifact is missing, Day 30 has produced no runtime NetworkPolicy evidence and mesh enrollment must stop.

The baseline’s gap is not “Kubernetes has no networking.” It is that Service routing and NetworkPolicy solve different layers from workload-authenticated encrypted transport and application-aware authorization.

Before this lesson, the team had manifests but no falsifiable traffic contract, so a later mesh change could be blamed or praised without comparison. Afterward, one successful sealed run connects the same customer request to Services, endpoints, identities, policies, two positive controls, and one exact denied timeout.

A mesh added without before-state evidence can hide a regression behind new green objects. Adding one is justified only by a measured gap, not by a component checklist.

• Freeze the customer path, Service endpoints, ServiceAccounts, and NetworkPolicy behavior before migration.
• A Service stabilizes changing Pod backends; NetworkPolicy constrains reachability when the network implementation enforces it.
• Static manifests do not prove enforcement; only the checked-in pre-mesh probe can seal this lesson’s allowed and denied runtime evidence.
• Ambient ztunnel and an optional waypoint add transport identity, encryption, telemetry, and policy layers; they do not replace application authorization.
• Reject enrollment when the approved path, denied path, or source-to-runtime mapping lacks fresh evidence.

The baseline is complete only when another operator can reproduce both outcomes from the same revision.

• [ ] Five Services select ready same-named workloads on port 8080
• [ ] Five Deployments use five same-named ServiceAccounts
• [ ] Pinned Kind/kindnet substrate and all four NetworkPolicies are recorded
• [ ] Both positive controls pass before and after evidence-to-generator returns exact timeout status 42
• [ ] Five new probe artifacts, revision, context, four Pod identities, time, and run ID are sealed