K8sSecDaP

architecture — k8ssecdap

Four-layer architecture. Layer one, supply chain: a git push triggers a Tekton pipeline, Kaniko builds the image rootlessly, Cosign signs it and attaches in-toto SLSA provenance, and the signed image plus attestation land in Harbor. Layer two, GitOps and admission: a Helmfile repository holds desired state, Argo CD reconciles it, and Kyverno verifyImages in Enforce mode checks the Cosign signature against Harbor before the API server admits the workload. Layer three, the cluster: three bare-metal nodes, one control plane and two workers, with Calico, MetalLB and Tailscale, plus platform services cert-manager, Sealed Secrets, Prometheus, Grafana and Loki. Layer four, runtime detection: an eBPF collector DaemonSet with a kprobe on tcp_v4_connect feeds a C++17 detection engine, which publishes over NATS to a Go aggregator, which fans out to incident-service on PostgreSQL, alert-bridge, MinIO evidence storage and the web console with Grafana.

A platform-engineering thesis with one thesis statement: nothing runs on the cluster that the cluster cannot prove it built. Every layer below exists to make that claim enforceable rather than aspirational.

The umbrella repository stitches six submodules together — infra, deploy, soc, pipeline, collector, report — so the whole platform is reproducible from a single git clone --recurse-submodules.

Layer 1 — the cluster, from bare metal

Three physical nodes, joined over a Tailscale mesh so the control plane keeps a stable address without a public IP anywhere. Ansible roles take a node from a bare reset to Ready: kernel tuning and cgroup setup, containerd, kubeadm bootstrap, Calico CNI, then MetalLB so type: LoadBalancer actually resolves to something on bare metal.

The roles are idempotent, which is the part that mattered in practice. Re-running the playbook after a failed join does not require thinking about which half-finished state the node is in.

Layer 2 — supply chain (the SLSA L3 part)

Builds run on Tekton inside the cluster. Kaniko does the image build with no Docker socket, no privileged container, no builder that could tamper with the artifact it is producing. Cosign signs the image and attaches in-toto provenance describing the source commit and the builder identity.

Kyverno enforces the other end: an admission policy verifies the Cosign signature against the expected key and rejects pods whose images cannot be attested. The gap between “we sign images” and “unsigned images cannot run” is where most supply-chain work quietly stops — closing it was the point.

Images land in a self-hosted Harbor registry, which also runs the Trivy scan.

Layer 3 — GitOps and platform services

Argo CD in App-of-Apps layout with auto-sync: the repository is the only way to change the cluster. Helmfile bootstraps the platform tier — cert-manager for certificates, Sealed Secrets so encrypted secrets are safe to commit, MetalLB, ingress — and Prometheus, Grafana and Loki for the observability stack.

Layer 4 — eBPF detection and the SOC

The detection path starts in the kernel. A custom eBPF collector attaches a kprobe to tcp_v4_connect and emits connection events through a ring buffer, built with libbpf and CO-RE so one binary works across kernel versions without recompiling per node.

Userspace hands events to a C++17 detection engine (zt-pipeline) that scores port-scan behaviour over sliding windows. Collector and engine ship together as a per-node DaemonSet.

Findings flow into a zero-trust SOC: an aggregator, an incident service, an alert bridge and a web console — Go and Python services over NATS, with PostgreSQL for incident state and MinIO for raw evidence, on Longhorn persistent volumes. Grafana dashboards sit on top for live scan activity.

Decisions and trade-offs

Every row here is a place where the obvious choice was cheaper and I took the other one on purpose. The third column is the part that matters.

Decision	The obvious alternative	Why, and what it cost
Kaniko for builds	Mount the Docker socket into the build pod	A builder holding the daemon socket can tamper with the artifact it is producing, which makes SLSA L3 provenance meaningless. Kaniko gives up layer-cache convenience; builds are slower and I stopped caring.
Kyverno in Enforce	Audit, and read the violations later	Audit mode produces a dashboard nobody acts on. Enforce means a bad deploy fails loudly at admission instead of quietly at 3 a.m. The cost is real: the admission webhook is now an availability dependency, so its failurePolicy, replica count and resource requests are load-bearing.
Bare metal + kubeadm	A managed control plane	The whole question is what happens below the API server — CNI, load balancing, kubelet, node lifecycle. A managed control plane hides exactly the layer under study. In exchange, every one of those layers became my problem.
Tailscale for the mesh	Public control-plane endpoint, or a VPN appliance	Stable addressing for nodes that have no public IP and sit behind a home router, with no port forwarding. The cost is a third-party service on the admin path — acceptable because nothing user-facing depends on it.
Sealed Secrets	Vault + External Secrets Operator	One controller, no extra HA service to keep alive, and encrypted secrets are safe to commit — which keeps GitOps actually complete. The cost is manual rotation and per-cluster re-encryption; on a real team I would pay for Vault instead.
Self-hosted Harbor	A hosted registry	Signature verification and the Trivy scan stay inside the same trust boundary as the cluster that enforces them. The cost is one more stateful service with backups I own.
Detection in eBPF	Parsing NetFlow or reading conntrack	A kprobe on tcp_v4_connect sees the connection attempt at the moment the kernel makes it, before anything can be dropped or rewritten. The cost is a build toolchain most people cannot debug, which is why CO-RE mattered — one binary, no per-kernel rebuild.

What I would change

The C++ engine and the Go SOC services split responsibility along a line that made sense while writing the thesis and less sense afterwards; window scoring probably belongs closer to the collector, with the SOC doing only correlation and case management.