Service Traffic Baseline: Why Add a Mesh?
Freeze the working Kubernetes traffic contract before changing how traffic is secured.
Run it in the public monorepo
This course is built around the public Zheta Kubernetes Lab monorepo. The excerpt below is runnable source, not pseudocode.
Source: scripts/probe-network-policy.sh
set +e
kubectl -n "$namespace" exec pod/"$DENIED_SOURCE_POD" -- env TARGET_IP="$target_ip" python -c 'import os,socket,sys
try:
socket.create_connection((os.environ["TARGET_IP"],8080),timeout=3)
except TimeoutError as error:
print(error,file=sys.stderr); raise SystemExit(42)
except Exception as error:
print(error,file=sys.stderr); raise SystemExit(43)
raise SystemExit(0)' 2> "$NETWORK_POLICY_EVIDENCE_DIR/negative-stderr.txt"
negative_status=$?
set -e
[ "$negative_status" = 42 ] || { echo "NetworkPolicy probe inconclusive: expected enforced timeout status 42, got $negative_status" >&2; exit 1; }Code to reality
- Declared intent
- Establish a live pre-mesh baseline by requiring the reviewed evidence Pod to time out against the exact generator Pod IP.
- Interpreter
- The shell and kubectl execute a bounded socket attempt; the pinned kindnet NetworkPolicy controller enforces the denied path.
- Software effect
- Only exact timeout status 42 is accepted; success, refusal, DNS failure, or another exception makes the probe inconclusive.
- Hardware effect
- The source Pod, destination node network stack, and node-local kindnet agent spend CPU and packets on the denied connection attempt.
- Observable evidence
- Reviewed Pod UIDs, kindnet status, exact policies, two positive controls before/after, and captured timeout stderr prove the baseline.
Start with the people and the result they need
The source tables below remain the detailed contract. Begin with these customer paths:
- D30-UC-01
- Person: Enterprise builder
- Job: Publish through the Forge control plane while its internal services remain reachable by stable names
- Observable result: The request enters control-plane:8080 and the required internal Service endpoints are ready
- D30-UC-02
- Person: Platform operator
- Job: Record the traffic and identity contract before adding a mesh
- Observable result: Source revision, five Services, five ServiceAccounts, ready workloads, EndpointSlices, and four NetworkPolicies agree
Turn each customer job into a testable story
Now turn each customer job into a story with a result that an engineer can check:
- D30-US-01
- Story: As an enterprise builder, I want publishing to use stable internal service names, so that Pod replacement does not change my workflow
- Observable acceptance: The same request succeeds through control-plane:8080; Service selectors have ready endpoints; the denied probe still fails
- D30-US-02
- Story: As a platform operator, I want an exact pre-mesh traffic map, so that I can detect migration regressions instead of guessing
- Observable acceptance: Evidence names Git revision, namespace, Service, port, selected Pods, ServiceAccount, NetworkPolicy, expected result, observed result, context,…
Add real state and observable proof
Finally trace each story through the system that owns its state and the evidence that proves the outcome:
- D30-FLOW-01
- Trigger: Builder sends an authenticated publish request to the control-plane Service
- Responsible systems: Kubernetes Service routing; control-plane, generator, runtime, evidence, and broker workloads; enforcing pod network
- Authoritative state: Git manifests plus Kubernetes API objects and EndpointSlices
- Owned record: ServiceTrafficContract
- Observable evidence: Request result, Service and EndpointSlice names, Pod UIDs, ServiceAccounts, policies, context, source revision, time, and trace or run ID
- Failure signal: Failed publish, empty endpoint set, or an approved TCP path denied
- D30-FLOW-02
- Trigger: Operator probes an undeclared direct path to an internal service
- Responsible systems: Kustomize, Kubernetes API server, Deployment controllers, Service proxy, and NetworkPolicy implementation
- Authoritative state: Git is desired configuration; Kubernetes API is observed object state; evidence record stores the comparison
- Owned record: WorkloadIdentityContract
- Observable evidence: Denied source and destination, port, policy, observed failure, unaffected positive control, context, source revision, time, and run ID
- Failure signal: Selector drift, unexpected allowed path, missing policy enforcement, or stale evidence
The enterprise problem and today’s slice
Enterprise problem: A service mesh can appear to improve security while silently breaking a customer publish request, because the team never recorded which service names, ports, workload identities, and NetworkPolicies already made that request work. Whole-course context: The incoming Forge application is rendered from Git into a local Kubernetes cluster; today captures its pre-mesh traffic contract before any Istio component or namespace label is added. Today’s slice: We inspect the five Services, five Deployments, five ServiceAccounts, and four effective NetworkPolicies, then run the checked-in kindnet positive/negative packet probe before explaining the remaining identity, encryption, and application-aware policy gap. End-of-day evidence: Only a successful probe may seal a source-revision-bound baseline containing reviewed Pod names and UIDs, kindnet status, exact policies, two allowed controls before and after, and one exact denied timeout. Still unsolved: Installing Istio, enrolling zheta-forge, mutual TLS, waypoints, mesh authorization, telemetry, failure recovery, and rollback remain deferred.
Customer use cases
Without a baseline, the application owner cannot tell whether a later failure came from the application, Kubernetes networking, or the mesh. A baseline is a before-state with both an allowed request and a denied request.
| Use case ID | Actor | Customer job | Success outcome | Denial or recovery evidence |
|---|---|---|---|---|
| D30-UC-01 | Enterprise builder | Publish through the Forge control plane while its internal services remain reachable by stable names | The request enters control-plane:8080 and the required internal Service endpoints are ready | A disallowed pod path is denied while the approved publish path remains healthy |
| D30-UC-02 | Platform operator | Record the traffic and identity contract before adding a mesh | Source revision, five Services, five ServiceAccounts, ready workloads, EndpointSlices, and four NetworkPolicies agree | Missing endpoint, wrong selector, unexpected caller, or policy mismatch blocks mesh enrollment |
Actor-centred user stories
A green Pod alone does not prove that a customer request can cross every required boundary. These stories join the customer action to network and identity evidence.
| Story ID | Use case IDs | User story | Observable acceptance conditions |
|---|---|---|---|
| D30-US-01 | D30-UC-01 | As an enterprise builder, I want publishing to use stable internal service names, so that Pod replacement does not change my workflow | The same request succeeds through control-plane:8080; Service selectors have ready endpoints; the denied probe still fails |
| D30-US-02 | D30-UC-02 | As a platform operator, I want an exact pre-mesh traffic map, so that I can detect migration regressions instead of guessing | Evidence names Git revision, namespace, Service, port, selected Pods, ServiceAccount, NetworkPolicy, expected result, observed result, context, time, and run ID |
End-to-end product flows
A customer sees one publish action, but Kubernetes must resolve several replaceable Pods behind stable Services. A Kubernetes Service supplies a long-lived address for changing backends, while EndpointSlices record the selected backends (Kubernetes networking).
| Flow ID | Use case IDs | Path | Trigger | Numbered steps | Terminal evidence |
|---|---|---|---|---|---|
| D30-FLOW-01 | D30-UC-01 | Happy | Builder sends an authenticated publish request to the control-plane Service | 1. Resolve control-plane. 2. Route TCP port 8080 to a ready control-plane Pod. 3. Resolve required generator, runtime, evidence, or broker Service names. 4. NetworkPolicy admits only declared Layer 3/4 paths. 5. Return the publish result. | Request result, Service and EndpointSlice names, Pod UIDs, ServiceAccounts, policies, context, source revision, time, and trace or run ID |
| D30-FLOW-02 | D30-UC-01, D30-UC-02 | Denied | Operator probes an undeclared direct path to an internal service | 1. Record the approved path as a positive control. 2. Send the bounded disallowed probe. 3. Observe timeout or rejection. 4. Re-run the approved path. | Denied source and destination, port, policy, observed failure, unaffected positive control, context, source revision, time, and run ID |
The smallest complete path is the builder’s action and its response. The diagram starts there because mesh work is safe only when the same customer evidence can be compared before and after the change.
System design derived from the flows
The baseline can control IP- and port-level reachability, but it cannot by itself prove workload identity or encrypt application traffic. Kubernetes also warns that a NetworkPolicy object has no effect when the cluster’s network implementation does not enforce it (Kubernetes networking).
| Use case ID | Entry point | Responsible services | Authoritative store | Failure evidence |
|---|---|---|---|---|
| D30-UC-01 | control-plane:8080 | Kubernetes Service routing; control-plane, generator, runtime, evidence, and broker workloads; enforcing pod network | Git manifests plus Kubernetes API objects and EndpointSlices | Failed publish, empty endpoint set, or an approved TCP path denied |
| D30-UC-02 | Git revision and read-only kubectl inspection | Kustomize, Kubernetes API server, Deployment controllers, Service proxy, and NetworkPolicy implementation | Git is desired configuration; Kubernetes API is observed object state; evidence record stores the comparison | Selector drift, unexpected allowed path, missing policy enforcement, or stale evidence |
The existing action and result keep their meaning. The added components show responsibility: Service selection finds Pods, workload processes consume node CPU and memory, and the network implementation admits or drops packets according to the policies.
Data model and ownership
Traffic evidence becomes unreliable when desired configuration, observed endpoints, and test results are mixed into one mutable note. Keep each owner explicit and correlate them by source revision and run ID.
Generated-application database: Not created in this slice — Kubernetes objects and a durable baseline evidence record are sufficient to compare transport behavior; customer domain data remains owned by the existing application stores.
| Record or entity | Store and owner | Primary key | Foreign key or opaque reference | Tenant key | Material constraint | Lifecycle and deletion | Use case IDs |
|---|---|---|---|---|---|---|---|
| ServiceTrafficContract | Git services.yaml and config.yaml, owned by Forge platform team | namespace plus Service name | Selector and named target-port reference to workload labels | zheta-forge namespace — this slice has one bounded workload tenant | Five names select their matching workload label and expose port 8080 to named target port http | Review in Git, promote by revision, replace through review, retain history | D30-UC-01, D30-UC-02 |
| WorkloadIdentityContract | Git workloads.yaml, owned by Forge platform team | namespace plus ServiceAccount or Deployment name | Deployment serviceAccountName and Pod-label reference | zheta-forge namespace — this slice has one bounded workload tenant | Each of five Deployments uses its same-named ServiceAccount and restrictive container settings | Create by reconciliation, update by revision, remove with workload after evidence retention | D30-UC-01, D30-UC-02 |
| NetworkPolicyContract | Git network-policy.yaml, owned by Forge security team | namespace plus NetworkPolicy name | Pod-selector and peer-selector references | zheta-forge namespace — policies do not grant cross-tenant application authority | Default deny is complemented only by named control-plane, internal-service, and broker paths | Review, reconcile, test allowed and denied paths, retain Git history, delete with namespace | D30-UC-01, D30-UC-02 |
| TrafficBaselineRun | Evidence artifact store, owned by platform operations | run_id | Opaque Git revision, context, object UID, request, and probe references | environment plus zheta-forge namespace | Both approved and denied paths must be fresh and from one context | Create before migration, seal for comparison, retain through rollback window, expire by evidence policy | D30-UC-01, D30-UC-02 |
The data layer adds four separately owned records without changing the request path. A later mesh result can now be compared with a sealed run instead of with memory.
Read the source contract exactly
A mesh migration can fail before any mesh component is involved if a Service selector, port, or identity is already wrong. These excerpts are exact slices of gitops/apps/forge/base/services.yaml, workloads.yaml, and network-policy.yaml in the implementation repository.
apiVersion: v1
kind: Service
metadata: { name: control-plane, namespace: zheta-forge }
spec: { selector: { app.kubernetes.io/name: control-plane }, ports: [{ name: http, port: 8080, targetPort: http }] }
| Effect field | What happens |
|---|---|
| Declared intent | Give control-plane Pods a stable in-cluster name and port |
| Interpreter | Kubernetes API server, EndpointSlice controller, and the cluster’s Service proxy implementation |
| Software effect | The Service selector tracks ready backends with label app.kubernetes.io/name: control-plane |
| Hardware effect | Node networking moves packets to the selected Pod; the process uses Pod CPU and memory |
| Evidence | kubectl get service,endpointslice shows the Service and ready backend addresses |
apiVersion: v1
kind: ServiceAccount
metadata: { name: control-plane, namespace: zheta-forge }
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: control-plane, namespace: zheta-forge }
spec:
replicas: 0
selector: { matchLabels: { app.kubernetes.io/name: control-plane } }
template:
metadata: { labels: { app.kubernetes.io/name: control-plane, app.kubernetes.io/part-of: zheta-forge } }
spec:
serviceAccountName: control-plane
| Effect field | What happens |
|---|---|
| Declared intent | Bind control-plane Pods to the control-plane Kubernetes ServiceAccount; the base deliberately runs zero replicas |
| Interpreter | Kustomize, Kubernetes API server, Deployment controller, scheduler, kubelet, and container runtime |
| Software effect | The local overlay replaces every Deployment replica count with one; the base alone starts no process |
| Hardware effect | Each local Pod consumes requested node CPU and memory and opens container port 8080 |
| Evidence | kubectl get deployment,pod,serviceaccount shows desired replicas, Pod UIDs, readiness, and the bound identity name |
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny, namespace: zheta-forge }
spec: { podSelector: {}, policyTypes: [Ingress, Egress] }
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: broker-publish-subscribe, namespace: zheta-forge }
spec:
podSelector: { matchLabels: { app.kubernetes.io/name: broker } }
policyTypes: [Ingress]
ingress:
- from: [{ podSelector: { matchExpressions: [{ key: app.kubernetes.io/name, operator: In, values: [control-plane, evidence] }] } }]
ports: [{ protocol: TCP, port: 8080 }]
| Effect field | What happens |
|---|---|
| Declared intent | Isolate every Pod by default, then admit broker ingress from control-plane and evidence Pods on TCP 8080 |
| Interpreter | Kubernetes API server and a NetworkPolicy-capable cluster network implementation |
| Software effect | Selected packet paths become allowed or denied at Layer 3/4; no TLS certificate or HTTP rule is created |
| Hardware effect | Node packet processing admits or drops traffic before the destination process handles it |
| Evidence | An allowed broker probe succeeds and a probe from an undeclared Pod fails, while policy objects alone are not accepted as runtime proof |
Capture a reproducible before-state
Reading YAML cannot prove runtime behavior, and changing policy before collecting evidence destroys the comparison. Render and deploy the local overlay, review the exact control-plane, evidence, broker, and generator Pods, then run the repository’s checked-in positive/negative probe before any Ambient label exists.
git rev-parse HEAD
kubectl kustomize gitops/apps/forge/overlays/local >/tmp/forge-baseline.yaml
kubectl apply -f /tmp/forge-baseline.yaml
kubectl -n zheta-forge rollout status deployment --all --timeout=180s
ALLOWED_SOURCE_POD="$(kubectl -n zheta-forge get pods -l app.kubernetes.io/name=control-plane -o jsonpath='{.items[0].metadata.name}')"
ALLOWED_SOURCE_POD_UID="$(kubectl -n zheta-forge get pod "$ALLOWED_SOURCE_POD" -o jsonpath='{.metadata.uid}')"
DENIED_SOURCE_POD="$(kubectl -n zheta-forge get pods -l app.kubernetes.io/name=evidence -o jsonpath='{.items[0].metadata.name}')"
DENIED_SOURCE_POD_UID="$(kubectl -n zheta-forge get pod "$DENIED_SOURCE_POD" -o jsonpath='{.metadata.uid}')"
DENIED_SOURCE_CONTROL_POD="$(kubectl -n zheta-forge get pods -l app.kubernetes.io/name=broker -o jsonpath='{.items[0].metadata.name}')"
DENIED_SOURCE_CONTROL_POD_UID="$(kubectl -n zheta-forge get pod "$DENIED_SOURCE_CONTROL_POD" -o jsonpath='{.metadata.uid}')"
TARGET_POD="$(kubectl -n zheta-forge get pods -l app.kubernetes.io/name=generator -o jsonpath='{.items[0].metadata.name}')"
TARGET_POD_UID="$(kubectl -n zheta-forge get pod "$TARGET_POD" -o jsonpath='{.metadata.uid}')"
NETWORK_POLICY_EVIDENCE_DIR="$(mktemp -d "${TMPDIR:-/tmp}/forge-network-policy.XXXXXX")"
export ALLOWED_SOURCE_POD ALLOWED_SOURCE_POD_UID DENIED_SOURCE_POD DENIED_SOURCE_POD_UID
export DENIED_SOURCE_CONTROL_POD DENIED_SOURCE_CONTROL_POD_UID TARGET_POD TARGET_POD_UID NETWORK_POLICY_EVIDENCE_DIR
printf 'review allowed=%s/%s denied=%s/%s control=%s/%s target=%s/%s\n' "$ALLOWED_SOURCE_POD" "$ALLOWED_SOURCE_POD_UID" "$DENIED_SOURCE_POD" "$DENIED_SOURCE_POD_UID" "$DENIED_SOURCE_CONTROL_POD" "$DENIED_SOURCE_CONTROL_POD_UID" "$TARGET_POD" "$TARGET_POD_UID"
NETWORK_POLICY_PROBE_APPROVED=1 make cni-probe
Approval is valid only after reviewing the printed names and immutable UIDs. make cni-probe invokes scripts/probe-network-policy.sh, which first runs scripts/verify-kind-network-policy.sh. That verifier requires the pinned Kind version, exact kind-$CLUSTER_NAME context, expected kindnet image, one ready kindnet agent per node, and ready nodes. The probe then vetoes non-local or already Ambient-labelled namespaces and refuses ambiguous, unready, wrongly owned, or changed Pods.
The experiment resolves the exact generator Pod IP. It proves control-plane-to-generator and evidence-to-broker reachability, attempts evidence-to-generator directly, and accepts only the script’s timeout status 42 as the denied result. Connection success, refusal, DNS failure, or any other status is inconclusive and fails the run. Both allowed controls run again afterward, and all four Pod identities are revalidated.
Do not claim a baseline merely because the command was printed or because four NetworkPolicy objects exist. A successful run creates new identities.txt, network-policies.yaml, positive-controls.txt, negative-stderr.txt, and kindnet-status.txt files in the pre-existing absolute evidence directory. Seal those artifacts with the source revision, context, time, and run ID. If the command does not complete successfully or any artifact is missing, Day 30 has produced no runtime NetworkPolicy evidence and mesh enrollment must stop.
Why the baseline still needs a mesh
The baseline’s gap is not “Kubernetes has no networking.” It is that Service routing and NetworkPolicy solve different layers from workload-authenticated encrypted transport and application-aware authorization.
| Need | Baseline owner | What remains unproved | Decision rule |
|---|---|---|---|
| Stable destination | Kubernetes Service and EndpointSlice | Caller identity, encryption, HTTP intent | Keep Services; do not ask a mesh to replace discovery |
| Reachability segmentation | Enforcing NetworkPolicy implementation | Cryptographic workload identity and mutual authentication | Keep NetworkPolicy as defense in depth; fail the migration if denied-path behavior weakens |
| Workload name | Kubernetes ServiceAccount reference | Peer presented that identity on this connection | Add a mesh only when identity evidence is required |
| Application policy | Forge application authorization | Uniform transport policy or waypoint-enforced HTTP method/path rules | Keep domain authorization in the app; use mesh policy only for transport and named traffic boundaries |
Istio ambient mode uses a per-node Layer 4 proxy called ztunnel for secure workload connectivity; an optional waypoint adds Layer 7 processing without adding sidecars to application Pods (Istio ambient overview). That makes a mesh complementary to, not a replacement for, Services, NetworkPolicy, or application authorization.
Before and after, side by side
Before this lesson, the team had manifests but no falsifiable traffic contract, so a later mesh change could be blamed or praised without comparison. Afterward, one successful sealed run connects the same customer request to Services, endpoints, identities, policies, two positive controls, and one exact denied timeout.
Key takeaways
A mesh added without before-state evidence can hide a regression behind new green objects. Adding one is justified only by a measured gap, not by a component checklist.
- Freeze the customer path, Service endpoints, ServiceAccounts, and NetworkPolicy behavior before migration.
- A Service stabilizes changing Pod backends; NetworkPolicy constrains reachability when the network implementation enforces it.
- Static manifests do not prove enforcement; only the checked-in pre-mesh probe can seal this lesson’s allowed and denied runtime evidence.
- Ambient ztunnel and an optional waypoint add transport identity, encryption, telemetry, and policy layers; they do not replace application authorization.
- Reject enrollment when the approved path, denied path, or source-to-runtime mapping lacks fresh evidence.
Checklist
The baseline is complete only when another operator can reproduce both outcomes from the same revision.
- [ ] Five Services select ready same-named workloads on port 8080
- [ ] Five Deployments use five same-named ServiceAccounts
- [ ] Pinned Kind/kindnet substrate and all four NetworkPolicies are recorded
- [ ] Both positive controls pass before and after evidence-to-generator returns exact timeout status 42
- [ ] Five new probe artifacts, revision, context, four Pod identities, time, and run ID are sealed