07

Cilium & eBPF

eBPF lets you load small, verified programs into the running Linux kernel and attach them to hooks — sockets, the traffic-control layer, the NIC driver. Cilium uses that to replace kube-proxy's iptables chains with hash-map lookups, to load-balance at connect() instead of per packet, and to enforce policy on workload identity rather than IP addresses. All of it ships as a DaemonSet; no kernel modules — which is exactly what an immutable OS like Talos wants.

iptables vs eBPF · Service lookup at scale

iptables (kube-proxy)KUBE-SERVICES: 447 rules, scanned top-down
0 rules checked
eBPF (Cilium)hash map: key = (VIP, port, proto)
hash → bucket 38 · 1 lookup
iptables nat rules
4,927
eBPF map entries
1,788
iptables evals / new conn
227
eBPF lookups / new conn
2
match cost per new connection vs #services (log y)
iptables O(n)nftables verdict mapeBPF hash O(1)10 svc20k svc

Only the first packet of a connection walks the chain — conntrack handles the rest — so the pain shows up as connection-setup latency, CPU on busy nodes, and slow rule syncs when Services churn. kube-proxy's nftables mode (GA in Kubernetes 1.33) also fixes the O(n) match with verdict maps; eBPF additionally bypasses netfilter altogether, keeping its own connection tracking in BPF maps.

socket LB

Load-balance once, at the socket

kube-proxy translates the ClusterIP on every packet. Cilium hooks the connect() syscall itself: the socket is pointed at a real backend before the first byte is sent.

Per-packet NAT vs socket-level load balancing

app
socket hook
tc / netfilter
wire
backend
■ dst 10.96.120.15 (ClusterIP)■ dst 10.244.2.7 (backend)
packets sent
0
NAT translations
0
NAT where
connect()
conntrack for NAT
not needed

A BPF program attached to the cgroup's connect() / sendmsg() hooks picks a backend and rewrites the socket's destination once. Every packet is born addressed to the pod; no per-packet DNAT, no reverse NAT on replies. getpeername() is translated back, so the app still believes it talks to the ClusterIP.

hooks

Where the programs attach

Different traffic is handled at different depths of the stack. The earlier the hook, the less work the kernel does — XDP runs in the NIC driver before the kernel has even allocated a packet buffer, which is why it's used to accelerate NodePort/LoadBalancer traffic and drop floods cheaply.

The eBPF hook points Cilium uses

  1. Socket layercgroup BPF: connect4/6, sendmsg, recvmsg, getpeername
  2. netfilter / iptablesbypassed by Cilium (kube-proxy lives here)
  3. tc (traffic control)bpf_lxc on each pod veth · bpf_host on eth0
  4. XDPin the NIC driver, before an skb is even allocated
  5. NICeth0 · hardware

Pod → ClusterIP is resolved at the socket, before a packet exists (socket LB). With sidecar meshes like Istio you set socketLB.hostNamespaceOnly=true so pods see the ClusterIP; translation then falls back to tc.

policy + observability

Identity, not IPs — and Hubble to watch it

Pod IPs churn constantly, so IP-based firewall rules are always out of date. Cilium hashes each pod's relevant labels into a numeric security identity, shared cluster-wide. Policy maps on every node are keyed on identities; packets carry the source identity (in the tunnel header, or resolved from the IP cache). Scale a Deployment and the new pods reuse the identity — nothing recompiles. Hubble taps the same datapath and records every flow with its verdict.

Security identities → policy verdicts → Hubble flow log

endpoints → security identities
ID 2reserved:world
any external IP
ID 48211k8s:app=frontendk8s:io.kubernetes.pod.namespace=shop
frontend… 10.244.1.17frontend… 10.244.2.31
ID 51002k8s:app=backendk8s:io.kubernetes.pod.namespace=shop
backend… 10.244.1.22backend… 10.244.3.8
ID 23817k8s:app=dbk8s:io.kubernetes.pod.namespace=shop
db… 10.244.3.12
ID 60433k8s:app=debugk8s:io.kubernetes.pod.namespace=shop
debug… 10.244.2.44
New pods reuse their label set's identity.
the policy for backend (frontend may GET /api/* on 8080)
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata: { name: backend, namespace: shop }
spec:
  endpointSelector: { matchLabels: { app: backend } }
  ingress:
    - fromEndpoints: [{ matchLabels: { app: frontend } }]
      toPorts:
        - ports: [{ port: "8080", protocol: TCP }]
          rules:
            http: [{ method: GET, path: "/api/.*" }]   # L7, via Envoy
$ hubble observe -n shop --follow
waiting for flows…
send→expect DROPPED
LB IPAM · L2 · BGP

LoadBalancer Services without a cloud

On bare metal or Proxmox nobody hands out LoadBalancer IPs. Cilium can do it itself: LB IPAM assigns addresses from a pool, and either L2 announcements (ARP from a lease-holding node) or the BGP control plane (advertise /32s to your router) makes them reachable — replacing MetalLB.

Announcing 192.168.1.240

click a node to fail / recover it
LAN clients / gatewayARP: who-has 192.168.1.240?worker-1 192.168.1.21standbycilium-agentworker-2 192.168.1.22lease holder: answers ARPcilium-agentworker-3 192.168.1.23standbycilium-agent

A type: LoadBalancer Service gets 192.168.1.240 from a CiliumLoadBalancerIPPool (LB IPAM). Then something has to tell the network where that IP lives.

L2 announcements: nodes race for a Kubernetes Lease per Service; the holder answers ARP for the VIP, so all traffic enters through one node (then eBPF spreads it to backends). Kill the leader: the lease expires and another node takes over and sends gratuitous ARP — a few seconds of disruption. Perfect for a homelab / Proxmox LAN with no BGP router.

192.168.1.240 reachable via worker-2
LB IPAM pool + L2 announcement policy
apiVersion: cilium.io/v2
kind: CiliumLoadBalancerIPPool
metadata: { name: lan }
spec:
  blocks:
    - start: 192.168.1.240
      stop: 192.168.1.250
---
apiVersion: cilium.io/v2alpha1
kind: CiliumL2AnnouncementPolicy
metadata: { name: lan }
spec:
  loadBalancerIPs: true
  interfaces: ["^eth[0-9]+", "^ens.*"]
  nodeSelector:
    matchExpressions:
      - { key: node-role.kubernetes.io/control-plane, operator: DoesNotExist }
…or BGP to a ToR router
# BGP control plane (v2 API; older releases used cilium.io/v2alpha1)
apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata: { name: tor }
spec:
  nodeSelector: { matchLabels: { bgp: "true" } }
  bgpInstances:
    - name: main
      localASN: 65001
      peers:
        - name: tor
          peerASN: 65000
          peerAddress: 192.168.1.1
          peerConfigRef: { name: tor }
# + CiliumBGPPeerConfig (timers, families) and
#   CiliumBGPAdvertisement { advertisementType: Service, service.addresses: [LoadBalancerIP] }
ecosystem

Beyond one cluster, beyond networking

Cluster Mesh
Connects multiple clusters' Cilium control planes (each exposes a clustermesh-apiserver). Pods get cross-cluster connectivity, identities are shared so policies can reference remote workloads, and a service.cilium.io/global: "true" annotation load-balances a Service across clusters for failover. Requires non-overlapping pod CIDRs and unique cluster IDs — plan this before you bootstrap.
Tetragon
A sibling project using the same eBPF machinery for runtime security: it observes process execution, file access, and syscalls with full Kubernetes context, and can enforce in-kernel (e.g. kill a process that opens /etc/shadow) via TracingPolicy resources. Installed separately with its own Helm chart.
helm values

Installing on Talos

The Talos-specific parts are the top block: let Talos own cgroups and kernel modules, reach the API via KubePrism, and take Pod CIDRs from the Kubernetes node objects. The rest is feature toggles.
cilium-values.yaml
# cilium-values.yaml — Talos (cluster.network.cni.name: none, cluster.proxy.disabled: true)
ipam:
  mode: kubernetes                 # use node.spec.podCIDR from Talos' podSubnets
kubeProxyReplacement: true
k8sServiceHost: localhost          # KubePrism: Talos' local API LB on every node
k8sServicePort: 7445
securityContext:
  capabilities:
    ciliumAgent: [CHOWN, KILL, NET_ADMIN, NET_RAW, IPC_LOCK, SYS_ADMIN, SYS_RESOURCE, DAC_OVERRIDE, FOWNER, SETGID, SETUID]
    cleanCiliumState: [NET_ADMIN, SYS_ADMIN, SYS_RESOURCE]
cgroup:
  autoMount: { enabled: false }    # Talos already mounts cgroup v2
  hostRoot: /sys/fs/cgroup

# optional extras
routingMode: tunnel                # or "native" + autoDirectNodeRoutes / BGP
tunnelProtocol: vxlan
bpf:
  masquerade: true
l2announcements:
  enabled: true                    # LB IPs on a flat LAN (homelab / Proxmox)
bgpControlPlane:
  enabled: false                   # or peer with your ToR instead
hubble:
  relay: { enabled: true }
  ui: { enabled: true }
encryption:
  enabled: false
  type: wireguard
gatewayAPI:
  enabled: false                   # needs the Gateway API CRDs installed first
install & verify
helm install cilium cilium/cilium --version 1.18.x -n kube-system -f cilium-values.yaml

cilium status --wait                       # agents, operator, KPR mode
cilium connectivity test                   # end-to-end conformance checks
kubectl -n kube-system exec ds/cilium -- cilium-dbg service list   # the eBPF service map
cilium hubble port-forward & hubble observe -n shop --verdict DROPPED
Kernel & capabilities
Talos ships a recent kernel with the eBPF features Cilium needs, but forbids loading kernel modules — that's why the default SYS_MODULE capability is removed from the agent's list.
  • why it matterseBPF swaps O(n) iptables chains for O(1) map lookups and moves Service translation to the socket, so load balancing costs one rewrite per connection instead of two per packet — and stays flat at thousands of Services.
  • why it mattersIdentity-based policy survives pod churn: labels map to a numeric identity, policy is enforced on identities, and scaling or rescheduling pods doesn't touch any rules. Hubble shows you each verdict as it happens.
  • why it mattersOn Talos, Cilium is a natural fit: set cni: none and disable kube-proxy, point it at KubePrism, and you also get LoadBalancer IPs (L2 or BGP), encryption and Gateway API without extra components.