The kernel knows everything except who you are

The detection half of my thesis needed one primitive: every outgoing TCP connection attempt made by a pod, labelled with which pod made it. Everything downstream — the scan scoring, the incident records, the network-policy drafts — is a function of that stream.

The first half of that sentence took an afternoon. The second half took a week.

Why tcp_v4_connect and not something easier

There are gentler places to get connection data:

• conntrack is already there, and already aggregated. That is the problem: by the time a flow appears in conntrack it has been established, and a port scan is mostly connections that never establish. Scanning a closed port produces a SYN and an RST, and conntrack is not where you go looking for that.
• NetFlow / IPFIX from the CNI gives you flows after sampling, after export delay, and without process identity. “Something in 10.244.2.0/24 touched 400 ports” is not an incident report anyone can act on.
• Hooking the connect() syscall works, but the syscall entry gives you a raw struct sockaddr __user * you must copy from user memory, and the syscall return tells you about the whole attempt including the eventual refusal.

tcp_v4_connect is the kernel function that actually initiates the connection. Hooking it means you see the attempt at the moment the kernel commits to it, before anything downstream can drop, rewrite or NAT it — and, importantly, before the peer has had a chance to refuse. For scan detection that ordering is the whole point: the signal is the attempt, not the outcome.

The kprobe and the kretprobe have to cooperate

Here is the catch that costs everyone their first afternoon. The signature is:

int tcp_v4_connect(struct sock *sk, struct sockaddr *uaddr, int addr_len)

At entry you have sk, but sk->__sk_common.skc_daddr is not populated yet — the function is what populates it. At return the sock is filled in, but a kretprobe does not receive the original arguments. Neither probe alone can see a complete event.

The fix is a small BPF hash map used as a per-task stash:

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __type(key, __u64);            // pid_tgid
    __type(value, struct sock *);
    __uint(max_entries, 4096);
} sock_store SEC(".maps");

SEC("kprobe/tcp_v4_connect")
int BPF_KPROBE(tcp_v4_connect_entry, struct sock *sk)
{
    __u64 pid_tgid = bpf_get_current_pid_tgid();
    bpf_map_update_elem(&sock_store, &pid_tgid, &sk, BPF_ANY);
    return 0;
}

pid_tgid is the right key because entry and return happen on the same task, and it is 64 bits of pid plus tgid, so two threads of one process do not collide.

There is a leak here I did not fix and should have: if the entry probe fires and the task dies before the return probe runs, the entry stays in the map forever. With max_entries: 4096 that eventually means bpf_map_update_elem starts failing and connections go unrecorded — silently, because nothing checks the return value. BPF_MAP_TYPE_LRU_HASH would evict the stale entries instead, at the cost of occasionally evicting a live one. That is the right trade and it is a two-line change.

-EINPROGRESS is not an error

The return probe filters on the result:

SEC("kretprobe/tcp_v4_connect")
int BPF_KRETPROBE(tcp_v4_connect_return, int ret)
{
    // 0 = connected, -115 = -EINPROGRESS (non-blocking socket, SYN is on the wire)
    if (ret != 0 && ret != -115)
        return 0;

Getting this wrong is the difference between a working collector and one that sees almost nothing. Any non-blocking socket — which is to say any scanner, any Go runtime, any Node process, anything using an event loop — returns -EINPROGRESS here. The SYN has been sent; the handshake simply has not finished. Filter to ret == 0 only and you have built a collector that watches blocking connect() calls, which in a modern cluster is close to nothing.

The subtle consequence in the other direction: a connection that will eventually be refused also passes this filter, because tcp_v4_connect returns before the RST comes back. That is exactly what a scan detector wants, and it is the second reason to hook here rather than at the syscall return.

Filtering in the kernel, because the ring buffer is the scarce resource

The threat model is a compromised pod, not a compromised node. So the source address is a filter, and the cheapest place to apply it is before the event ever costs a ring-buffer slot:

struct {
    __uint(type, BPF_MAP_TYPE_LPM_TRIE);
    __type(key, struct lpm_key);
    __type(value, __u32);
    __uint(max_entries, 1024);
    __uint(map_flags, BPF_F_NO_PREALLOC);   // mandatory for LPM_TRIE
} cidr_map SEC(".maps");

BPF_F_NO_PREALLOC is not optional there. Leave it out and the map simply fails to create, with an error that does not mention prealloc. An LPM trie gives longest-prefix matching in the kernel, so one lookup answers “is this address in the pod network, the service network, the node network, or none of the above” against arbitrary CIDRs.

struct lpm_key src_key = { .prefixlen = 32, .addr = src_ip };
__u32 *src_label = bpf_map_lookup_elem(&cidr_map, &src_key);
if (!src_label || *src_label != 1)   // 1 = pod network
    return 0;

On a busy node this drops the node’s own outbound traffic — kubelet talking to the API server, image pulls, the CNI’s own chatter — before it becomes a userspace problem. The ring buffer is 256 KB. Anything that does not need to cross it should not.

What I got lazy about: the CIDRs are hardcoded in userspace, 10.244.0.0/16 for pods and 10.96.0.0/12 for services, which is right for a kubeadm default and wrong for anyone else. They belong in the DaemonSet’s config, read from the cluster.

Then the actual problem

At this point the kernel hands userspace a clean event:

struct network_event_t {
    __u32 src_ip, dst_ip;  __u16 dst_port;
    __u32 pid, ppid, uid;
    __u64 timestamp_ns, cgroup_id;
    char  comm[16];
};

and none of it says pod. The kernel has no concept of a pod. It has cgroups, pids and network namespaces; “pod” is a fiction maintained entirely in userspace by the kubelet. An alert that says pid 41288 (comm "nmap") scanned 400 ports is true, and useless — by the time anyone reads it, pid 41288 does not exist.

The obvious answer is to ask the API server. I did not want to, for three reasons: it puts a per-event round trip in the hot path, it means every node runs a DaemonSet with cluster-wide pod read, and it fails exactly when you need it most — during an incident that is hammering the control plane.

So: recover the identity from /proc, offline.

/proc/<pid>/cgroup is four formats wearing a trench coat

3:cpu:/kubepods/besteffort/pod3f2a.../a1b2c3...              cgroup v1, cgroupfs driver
0::/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f2a_....slice/cri-containerd-a1b2c3....scope
                                                             cgroup v2, systemd driver

Between cgroup v1 and v2, the systemd and cgroupfs drivers, and the containerd / CRI-O / Docker prefixes, the parser has to handle the cross product. The container id is the last path segment, minus a .scope or .slice suffix, minus one of cri-containerd-, crio-, docker-, containerd-, and it is only a container id if what remains is at least 32 lower-hex characters.

The pod UID hides in an earlier segment, and this is where I lost an hour:

// Use rfind so we match the real "...-pod<UID>" segment and
// not the "pod" inside the parent "kubepods" slice.
auto pos = segment.rfind("pod");

find("pod") matches the pod in kube**pod**s-besteffort-pod3f2a..., and you get a “UID” of s-besteffort-pod3f2a.... It looks almost right, which is the worst kind of wrong. rfind plus a length and charset check on the result — a pod UID is a UUID, so hex and dashes, at least 32 characters — rejects both that and the pods in kubepods-besteffort.

Reading inside the container without entering it

The container id is not a pod name. But every pod has two files that are:

// namespace, from the pod's own mounted service-account token
base + "/root/var/run/secrets/kubernetes.io/serviceaccount/namespace"

// pod name, best-effort: a pod's hostname defaults to the pod name
base + "/root/etc/hostname"

/proc/<pid>/root is a symlink into that process’s mount namespace root. Opening a path through it reads the container’s filesystem, from the host, with no nsenter, no runtime socket and no API call. The service-account namespace file is authoritative. /etc/hostname is genuinely best-effort — it breaks for hostNetwork pods and for anything setting an explicit hostname or subdomain — so the code falls back to the pod UID, which is at least joinable against the API server later, by a human, at their leisure.

The race that makes caching mandatory

The gap between “the kernel emitted the event” and “userspace polls the ring buffer” is small but not zero, and a scanning process is often short-lived. Read /proc/<pid> too late and there is nothing there.

Caching by pid does not help — pids are recycled. The cache is keyed on cgroup_id, which the kernel gives us for free via bpf_get_current_cgroup_id() and which is stable for the lifetime of the container:

auto it = g_meta_cache.find(evt->cgroup_id);
if (it == g_meta_cache.end() || it->second.container_id.empty()) {
    ztmeta::PodMeta m = ztmeta::resolve_pod_meta(evt->pid);
    it = g_meta_cache.insert_or_assign(evt->cgroup_id, std::move(m)).first;
}

The container_id.empty() half of that condition is the part that matters. A cache entry that resolved to nothing is a failed lookup, not a negative result — the next event from the same container will have a different, possibly still-alive pid, so it is worth trying again. Caching the failure would mean one unlucky first event permanently un-labels a container.

The map grows without bound as containers churn, which on a node with frequent restarts is a slow leak. Evicting on cgroup removal would need an inotify watch on the cgroup hierarchy; a bounded LRU is the cheap version.

Testing the part that can actually be tested

You cannot unit-test a kprobe in CI. You can unit-test the parser, and the parser is where all the bugs were, so the pure functions are split out deliberately:

// The pure parsers (parse_cgroup_line) are split out so they can be unit-tested
// with fixture strings, no live /proc required.
bool parse_cgroup_line(const std::string& line,
                       std::string& container_id,
                       std::string& pod_uid);

Every cgroup format above exists as a fixture string in the test file. That is the whole trick: the privileged, kernel-dependent, root-only part of the program is small and mostly straight-line, and the fiddly part is a string function. Keeping the boundary between them sharp is what makes any of it testable on a laptop.

What I would change

• BPF_MAP_TYPE_LRU_HASH for sock_store, so a killed task cannot silently fill it.
• CIDRs from the DaemonSet config rather than compiled in.
• tcp_v6_connect is not hooked at all. On a dual-stack cluster half the connections are invisible, which is the kind of gap an attacker only has to find once.
• Pod name from the kubelet’s pod-resources socket rather than /etc/hostname — still node-local, still no API server, and correct for hostNetwork pods.

The recurring lesson is the one in the title. Getting data out of the kernel is a solved problem with good libraries and a lot of documentation. Attaching a name to that data — one that means something to the person who has to act on the alert at 3am — is where the work is, and none of it happens in the kernel.