Most DevSecOps tutorials end at "add a linting step to your GitHub Actions." This lab went further. Nine Multipass VMs on a 16GB laptop, four services in three languages, and every layer of a production security pipeline built from scratch: secrets management, container scanning, service mesh, zero-trust authorization, and event-driven architecture. If you are hiring for DevSecOps or platform-adjacent roles, this post is the honest breakdown of what I built, what broke, and what I would defend in a design review.
Motivation: Build What a Security Team Would Actually Run
The goal was not to follow a tutorial. It was to answer one question: what does a bank's internal platform look like from the inside? That means private registries, audited secret access, encrypted service communication, and pipelines that reject bad code before it touches a cluster, not after.
The lab had four services: a Java Spring Boot backend, a Go API, a Python FastAPI service, and a React frontend. Each had its own Jenkins pipeline, its own Helm chart, its own security posture. The infrastructure ran on nine Multipass VMs: jenkins, jenkins-node, nexus, harbor, sonarqube, k8s-master, k8s-worker, grafana, and vault.
[Placeholder: Architecture diagram showing all 9 VMs, their roles, and how they connect to the K8s cluster and CI/CD pipeline.]
The CI/CD Pipeline: Four Gates on Every Push
Jenkins multibranch pipelines run on every push across all four services. Nothing reaches the cluster without passing all four gates in sequence.
| Gate | Tool | What it catches |
|---|---|---|
| Secret scanning | Gitleaks | API keys, tokens committed by accident |
| Static analysis | Semgrep | Insecure code patterns, hardcoded credentials |
| Container CVE scan | Trivy | Vulnerable base images and dependencies |
| Code quality | SonarQube | Coverage thresholds, maintainability issues |
The ordering matters. Gitleaks runs first: a leaked secret is a hard stop regardless of code quality. Trivy runs last because it needs the built image. The pipeline fails fast on the cheapest checks and only spends time on expensive ones when the codebase has already proven it is not obviously broken.
Images are pushed to a private Harbor registry and Helm charts to a private Nexus repository. No public registries in the build chain. Every artifact is internal, versioned, and immutable.
[Placeholder: Screenshot of Jenkins pipeline showing all four security stages passing on a representative build.]
Secrets Management: Two Patterns for Two Problems
Hardcoded credentials in Jenkins are a liability. Secrets in Helm values are worse. HashiCorp Vault solved both with two distinct delivery patterns, because the consumers are fundamentally different.
Jenkins AppRole (build-time credentials): Jenkins authenticates to Vault using a role ID and secret ID. Every pipeline fetches kubeconfig, Harbor credentials, and Nexus credentials at build time. Jenkins stores nothing sensitive; a compromised Jenkins instance cannot read application secrets. Every credential fetch is logged in Vault's audit trail.
Kubernetes sidecar injection (runtime secrets): corona-go and corona-python pods run a vault-agent sidecar that authenticates via their Kubernetes ServiceAccount, fetches secrets from Vault, and writes them to a shared tmpfs volume the application reads at startup. No secrets in environment variables. No secrets in Helm values. Zero plaintext credentials anywhere in the deployment chain.
The two patterns solve different problems. Jenkins does not need runtime app secrets. Pods do not need deploy credentials. Mixing them would be architecturally wrong, not just untidy.
[Placeholder: Diagram showing the two Vault patterns side by side: AppRole on the left for Jenkins build flow, sidecar injection on the right for pod startup flow.]
Kafka + Redis: Async Over Tight Coupling
The four services originally communicated over synchronous HTTP. The question I kept asking was: what happens when one service goes down? The answer with HTTP is cascading failure. The answer with Kafka is nothing: the message waits.
corona-backend (Java) publishes events to a corona-events topic with 3 partitions, running in KRaft mode with no Zookeeper. corona-go and corona-python consume independently via separate consumer groups. Same message, two consumers, neither blocks the other. Redis caches counters and recent event state for sub-millisecond reads.
| Aspect | Kafka | Redis |
|---|---|---|
| Storage | Disk, durable, days/weeks | RAM, ephemeral |
| Replay | Yes, any consumer group | Only with Streams |
| Latency | ~5ms | ~0.5ms |
| Best for | Events, audit, history | Cache, counters, ephemeral state |
They are not alternatives. They solve different problems in the same architecture. The most common mistake in tutorials is treating them as competing options.
[Placeholder: Diagram of the event flow: backend publishes to Kafka, two consumer groups receive independently, Redis caches the counter state.]
Istio: Three Things Built on Top of the Mesh
Adding Istio to the cluster changed nothing in the application code. That is the point. Envoy sidecars were auto-injected into every pod in the default namespace: corona-go went from 1/1 to 3/3 (app + vault-agent + istio-proxy) without a single line changed in main.go. The mesh is invisible to the application.
mTLS (PERMISSIVE mode)
Service-to-service traffic is automatically upgraded to mutual TLS inside the mesh. External NodePort traffic still works in plain HTTP. STRICT mode was tested and rejected; it breaks NodePort access, which requires an IngressGateway to handle TLS termination at the edge. PERMISSIVE was the right choice for this topology, and I can explain why that is a deliberate architectural decision rather than a shortcut.
AuthorizationPolicy (zero-trust)
Deny-all by default. Every request in the mesh returns 403 until an explicit ALLOW rule permits it. The policy structure:
- Only
corona-backend's ServiceAccount identity can call corona-go and corona-python - Public endpoints (
/events,/health) are explicitly opened for monitoring - Everything else requires the correct identity, even from pods inside the mesh with valid mTLS certificates
The proof: same pod, same mTLS certificate, two different paths. GET / returns 403. GET /events returns 200. mTLS proves identity. AuthorizationPolicy decides permission. Two independent layers of zero-trust.
Blue-Green deployment
Two versions of corona-go ran simultaneously. A VirtualService routed production traffic to v1 (100%) and test traffic with an x-canary: v2 header to v2. Real users never saw the new version until cutover: a single VirtualService edit, zero pod restarts, zero downtime.
The important lesson: VirtualService routing only applies to traffic originating from inside the mesh. NodePort traffic is handled by kube-proxy at L4 and bypasses Istio's routing entirely. Testing required a pod inside the mesh as the caller. This distinction matters in any real environment using an Ingress controller.
[Placeholder: Kiali screenshot showing the live mesh graph with mTLS lock icons on service-to-service edges, and the two corona-go versions visible as separate nodes.]
Fault Injection: Chaos Testing in the Mesh
A VirtualService fault rule injected a 5-second delay into 50% of HTTP requests to corona-go. Across 10 requests, roughly half took ~10ms and half took ~5000ms, exactly the injected distribution. The interesting result was what it did not affect: the Kafka event flow.
corona-backend published an event during the fault injection window. corona-go consumed it immediately, with no delay. Istio's L7 fault injection applies to HTTP routes. Kafka is TCP. The two protocols exist in entirely different layers of the stack. This is the concrete, demonstrable version of "async messaging decouples services from each other's failure modes." Not a theoretical claim: a measured result.
What Actually Broke (The Real Learning)
containerd + Harbor over HTTP. k8s-master's containerd kept attempting HTTPS despite a hosts.toml configuration pointing at HTTP. Every pod scheduled on master failed with "server gave HTTP response to HTTPS client." The permanent fix: nodeSelector: k8s-worker baked into every Helm chart. Master handles the control plane. Worker handles workloads. The containerd misconfiguration became irrelevant.
Vault + Istio port conflict. Istio's iptables rules intercept all outbound traffic from a pod, including vault-agent-init's connection to Vault on port 8200. Pods sat stuck at Init:1/2 indefinitely, with vault-agent-init logging "connection refused" against a Vault that was fully healthy. The fix: traffic.sidecar.istio.io/excludeOutboundPorts: "8200,8201" in pod annotations. One annotation, now in every Helm chart that uses Vault injection.
CoreDNS editing is dangerous. I tried adding vault.lab to the CoreDNS Corefile. A YAML syntax error in the Corefile crashed both CoreDNS pods and took down cluster DNS; everything dependent on name resolution stopped working simultaneously. The safer approach is hostAliases in the pod spec, which adds the entry to that specific pod's /etc/hosts without touching cluster-wide DNS. Targeted, no blast radius, survives CoreDNS restarts.
Helm boolean gotcha. {{ if .Values.flag | default true }} evaluates to true even when --set flag=false is passed. Helm's default function treats false as an empty value and substitutes the fallback. The fix: declare explicit defaults in values.yaml and reference the variable directly. This cost an hour and is now a permanent note in the troubleshooting docs.
DestinationRule and VirtualService are a matched pair. Deleting a DestinationRule while a VirtualService still references its subsets causes 503 "no healthy upstream." Envoy has routing rules pointing at subsets it can no longer resolve. Both resources must be created and deleted together. Learning this through a broken demo is the most memorable way to learn it.
Closing
The lab was a lesson in how much engineering lives outside the application code. The security pipeline, the secret delivery patterns, the service mesh, the routing rules: none of it appears in the services themselves. All of it is infrastructure, configured declaratively, applied and removed independently of deployments.
The stack is not a checklist chased for buzzwords. It is the minimum credible story for a security-conscious internal platform: pipelines that reject bad code before it touches the cluster, secrets that rotate without a redeploy, service communication that is encrypted by default, and deployment patterns that make rollback as easy as applying a YAML file.
If you are building something similar, understand the why before copying the how. PERMISSIVE vs STRICT mTLS is an architecture question, not a security slider. Kafka vs Redis is a use-case question, not a popularity contest. The tools are interchangeable. The reasoning is what persists.
Stack: Jenkins, Gitleaks, Semgrep, Trivy, SonarQube, HashiCorp Vault, Kubernetes, Helm, Kafka (KRaft), Redis, Istio, Envoy, Kiali, Prometheus, Grafana, Harbor, Nexus, running on 9 Multipass VMs, 16GB laptop, zero cloud bill.