CNIs compared

Kubernetes defines the network model but ships no implementation. A CNI plugin is the program the container runtime executes for every pod to wire it up: create the interface, give it an IP, make it routable. Everything else — tunnels vs BGP, iptables vs eBPF, NetworkPolicy, encryption — is the CNI's design choice. Talos ships Flannel; here's what it does, what the alternatives add, and how to swap.

What happens when a pod starts · the CNI contract

kubelet

CRI client

›

containerd

runtime

›

CNI plugin

/opt/cni/bin/*

›

IPAM

host-local

ADD · step 1/9

kubelet asks the runtime for a pod sandbox

The scheduler bound the pod to this node. kubelet calls the CRI RunPodSandbox RPC on containerd. Kubernetes itself never touches interfaces or IPs.

CRI gRPC

RunPodSandbox {
  config: { metadata: { name: "web-7d9f", namespace: "default" },
            linux: { ... } }
}

overlay vs routed

Encapsulation, MTU and the bytes you pay

Pod IPs aren't known to your physical network, so a CNI either tunnels them inside node-to-node packets or advertises them to the routers (BGP). Tunnels cost header bytes on every packet and require a smaller pod MTU — get that wrong and you get the most confusing outage in networking.

Encapsulation explorer

underlay MTU1500IPv6 underlay

header stack (zoomed: bar = 120 bytes)overhead 50 B

outer IPv420

UDP8

VXLAN8

inner Eth14

inner IP20

TCP20

payload →

■ outer IPv4: node → node addresses■ UDP: dst port 8472■ VXLAN: 24-bit VNI■ inner Eth: original L2 frame header

one full-size packet, true scale (1500 B underlay MTU)1500 B on the wire

Mistake: leave the pod MTU at 1500 (ignore overhead)

overhead

50 B

pod MTU

1450

TCP MSS

1410

payload efficiency

92.9%

VXLAN

Wraps the whole L2 frame in UDP. Works on any IP network, crosses routers and most cloud fabrics, and the UDP source port carries flow entropy for ECMP. Used by: Flannel (Talos default), Cilium default, Calico VXLAN mode.

comparison

The field

All of these implement the same CNI contract; they differ in dataplane, routing model and what they do beyond connectivity.

	Flannel	Cilium	Calico	Kube-router	Kube-OVN	Antrea	Multus
Dataplane	Linux bridge + routes	eBPF	iptables / nftables / eBPF	routes + iptables + IPVS	OVS (OVN)	OVS	meta-plugin (delegates)
Pod routing	VXLAN, host-gw, WireGuard	VXLAN / Geneve or native	BGP native, IPIP, VXLAN	BGP native (+IPIP)	Geneve, VLAN underlay	Geneve, VXLAN, noEncap	whatever it delegates to
NetworkPolicy	none	L3/L4 + L7, identity-based	L3/L4, tiers, global	L3/L4	L3/L4	L3/L4, tiers (+L7)	—
kube-proxy replacement	no	full (eBPF)	eBPF dataplane	IPVS proxy	partial	AntreaProxy	—
Encryption	WireGuard backend	WireGuard, IPsec	WireGuard	no	IPsec	IPsec, WireGuard	—
BGP	no	BGP control plane	native, full mesh / RR	GoBGP built in	LB/pod IP export	BGP policy	—
Observability	—	Hubble (flows, L7, UI)	flow logs, Whisker UI	—	mirroring, tracing	Theia, traceflow	—
On Talos	default, built in	cni: none + Helm	cni: none + operator	cni: none + manifest	cni: none + Helm	cni: none + Helm	add on top of primary

chooser

Pick your CNI

Tick what you actually need. A missing hard requirement disqualifies an option; ties go to whatever does more of what you asked.

Requirements → candidates

NetworkPolicyenforce the standard Kubernetes APIL7 policyallow GET /api but not POST /adminBGP to ToRadvertise pod / LB IPs to your routerskube-proxy replacementdrop iptables Service rulesEncryptionnode-to-node WireGuard / IPsecSimplicityfew moving parts, nothing to learnWindows nodesmixed Linux / Windows clusterMultiple NICs per podSR-IOV, storage or telco networksFlow observabilitysee who talks to whom, and verdicts

1.CiliumeBPF
4/4
NetworkPolicykube-proxy replacement
eBPF dataplane, full kube-proxy replacement, identity-based L3–L7 policy, Hubble, BGP control plane, LB IPAM, WireGuard/IPsec, Gateway API. Most capable; most to learn.
2.CalicoBGP + policy
4/4
NetworkPolicykube-proxy replacement
iptables, nftables or eBPF dataplane; native BGP routing or IPIP/VXLAN; tiered GlobalNetworkPolicy; WireGuard; strong Windows support. Recent releases add flow logs + the Whisker UI.
3.AntreaOVS
4/4
NetworkPolicykube-proxy replacement
Open vSwitch dataplane, Geneve by default, AntreaProxy can replace kube-proxy, tiered ClusterNetworkPolicy, IPsec/WireGuard, first-class Windows.
4.Kube-routerlean
4/4
NetworkPolicykube-proxy replacement
5.Kube-OVNSDN
3/4
NetworkPolicykube-proxy replacement
6.FlannelTalos default
0/4
NetworkPolicykube-proxy replacement

Scores are a rough guide: 2 = first-class, 1 = partial / add-on / limited, 0 = missing (struck through). A missing hard requirement sinks the option regardless of score.

machine config

Installing it on Talos

Talos owns the node, so the CNI choice lives in the machine config. Flannel needs nothing. For anything else, set cluster.network.cni.name: none before bootstrap; the nodes come up NotReady (no CNI config in /etc/cni/net.d) until you install one. With Cilium's kube-proxy replacement, also disable kube-proxy so two dataplanes don't fight over Services.

Flannel (default)

# Nothing to do: Talos deploys Flannel (VXLAN) out of the box.
# Optional tuning, e.g. pin the interface Flannel uses on multi-NIC nodes:
cluster:
  network:
    cni:
      name: flannel
      flannel:
        extraArgs:
          - --iface-can-reach=192.168.1.1

Cilium · machine config patch

# patch: cni-none.yaml — apply to ALL control plane nodes before bootstrap
cluster:
  network:
    cni:
      name: none          # Talos installs no CNI; nodes stay NotReady until you do
  proxy:
    disabled: true        # Cilium replaces kube-proxy

Cilium · Helm install with Talos-specific values

helm repo add cilium https://helm.cilium.io && helm repo update

helm install cilium cilium/cilium --version 1.18.x --namespace kube-system \
  --set ipam.mode=kubernetes \
  --set kubeProxyReplacement=true \
  --set securityContext.capabilities.ciliumAgent="{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}" \
  --set securityContext.capabilities.cleanCiliumState="{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}" \
  --set cgroup.autoMount.enabled=false \
  --set cgroup.hostRoot=/sys/fs/cgroup \
  --set k8sServiceHost=localhost \
  --set k8sServicePort=7445

Why those values?

Talos mounts cgroup v2 itself, so Cilium must not (cgroup.autoMount.enabled=false, hostRoot=/sys/fs/cgroup). Talos doesn't allow the SYS_MODULE capability (kernel modules are baked in), hence the explicit capability lists.

Why localhost:7445?

Without kube-proxy there's no ClusterIP for kubernetes.default yet — Cilium is what would provide it. KubePrism, Talos's on-node API-server load balancer, listens on localhost:7445 on every node, breaking the chicken-and-egg.

Bootstrap ordering

Helm can only run once the API is up, so nodes wait NotReady for a few minutes. To make it hands-off, render the chart with helm template into cluster.inlineManifests, or point extraManifests at a URL.

Calico · tigera operator

# Talos: same cni.name: none patch (keep kube-proxy unless you use the eBPF dataplane)
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.30.x/manifests/tigera-operator.yaml

kubectl apply -f - <<'EOF'
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  calicoNetwork:
    linuxDataplane: Nftables        # or Iptables / BPF
    bgp: Enabled
    ipPools:
      - cidr: 10.244.0.0/16           # must match cluster.network.podSubnets
        encapsulation: VXLANCrossSubnet   # routed within a subnet, VXLAN across
        natOutgoing: Enabled
EOF

why it mattersA CNI is a contract, not a product: runtime execs a binary with ADD/DEL, it builds a veth, asks IPAM for an IP, sets routes and returns JSON. Everything users notice — policy, encryption, Service speed — is extra.
why it mattersOverlay vs routed is a trade: tunnels work on any network but cost 20–60 bytes per packet and need a lower pod MTU; native routing is free but your fabric must learn pod routes (BGP or a flat L2).
why it mattersFlannel is a fine default until you need NetworkPolicy — it silently ignores policies. Choose the CNI before bootstrap; swapping a live cluster's dataplane is a migration, not a config change.