Bootstrap & etcd

Going from blank machines to a Kubernetes cluster on Talos takes three commands: apply a config to each node, bootstrap etcd on exactly one control plane, and fetch a kubeconfig. Everything in between (installing to disk, starting etcd, running the control-plane static pods, joining the other members) is done by machined on each node. This page walks through that sequence, then looks at the two things that decide whether the control plane stays up: etcd quorum and the API endpoint.

Cluster bring-up, step by step

3 control planes · 2 workers

mistake: also run bootstrap on cp-2

01/10Boot the ISO

Every machine boots the Talos ISO/PXE image into maintenance mode: no config, nothing installed, apid listening on :50000 waiting for one.

cp-1

10.0.0.10 · cp

cp-2

10.0.0.11 · cp

cp-3

10.0.0.12 · cp

worker-1

10.0.0.21 · worker

worker-2

10.0.0.22 · worker

disk

ISO · maintenance

config

no config

etcd

—

control plane

—

kubelet

—

Node

—

the whole bring-up

# 1. nodes booted from ISO are in maintenance mode
talosctl get disks --insecure -n 10.0.0.10      # pick the install disk

# 2. push configs (unauthenticated, only possible in maintenance mode)
for ip in 10.0.0.10 10.0.0.11 10.0.0.12; do
  talosctl apply-config --insecure -n $ip -f controlplane.yaml
done
for ip in 10.0.0.21 10.0.0.22; do
  talosctl apply-config --insecure -n $ip -f worker.yaml
done

# 3. once, on one control plane
talosctl bootstrap -n 10.0.0.10 -e 10.0.0.10

# 4. wait for everything, then grab a kubeconfig
talosctl health -n 10.0.0.10
talosctl kubeconfig -n 10.0.0.10

Why bootstrap only once?

talosctl bootstrap tells a node to create a new etcd cluster rather than join one. The other control planes discover the existing cluster and join it automatically. Run bootstrap on a second node (or again after losing quorum) and you get a separate cluster. To recover a cluster that lost etcd, use talosctl bootstrap --recover-from=db.snapshot on one node instead.

etcd, through the Talos API

talosctl -n 10.0.0.10 etcd members
talosctl -n 10.0.0.10 etcd status
talosctl -n 10.0.0.10 etcd snapshot db.snapshot     # back this up!
talosctl -n 10.0.0.12 etcd leave                    # before removing a CP
talosctl -n 10.0.0.10 etcd remove-member <id>       # if it died for good

Raft majority

etcd quorum: why 3 or 5, never 4

etcd is a Raft cluster: a write is committed only when a majority of members (⌊n/2⌋+1) has it. That majority is what keeps two halves of a partitioned cluster from both accepting writes. Adding a fourth member raises the quorum to 3 without tolerating any more failures, so even counts only add another thing that can break.

Quorum simulator

etcd members

Click a member to fail or recover it.

quorum needed

2 of 3

failures tolerated

healthy

3/3

writes (and K8s API)

available

A majority can elect a leader and commit writes. The Kubernetes API keeps working.

members	quorum	tolerates
1	1	0
2	2	0same as 1, more to fail
3	2	1
4	3	1same as 3, more to fail
5	3	2

cluster.controlPlane.endpoint

One address for the API

Every kubelet, every kubeconfig and every certificate refers to a single endpoint. It must keep working when any one control plane is down, so it can never be the IP of a single node. Talos gives you a built-in option (a shared VIP) and works with any external load balancer or DNS name. KubePrism then shields in-cluster clients from endpoint outages entirely.

Endpoint options

KubePrism on worker

cluster.controlPlane.endpoint: https://10.0.0.5:6443

Click a control plane to power it off.

goodBuilt in: no extra machines. Control planes elect an owner, which answers ARP for the VIP.

watch outAll CPs must share an L2 segment. The election goes through etcd, so the VIP can't move without quorum. Failover takes a few seconds or more. Don't use it as your talosctl endpoint.

KubePrism (on by default) runs a small TCP load balancer on every node at localhost:7445 that spreads connections over all known API servers. In-cluster, host-level clients keep working even if the external endpoint is down or the VIP is moving.

shared VIP (per control-plane patch)

machine:
  network:
    interfaces:
      - deviceSelector:
          physical: true
        dhcp: true
        vip:
          ip: 10.0.0.5        # unused IP on the same L2 segment
cluster:
  controlPlane:
    endpoint: https://10.0.0.5:6443

KubePrism (default in generated configs)

machine:
  features:
    kubePrism:
      enabled: true
      port: 7445
# e.g. Cilium with kube-proxy replacement:
#   k8sServiceHost: localhost
#   k8sServicePort: 7445

membership

How nodes find each other: discovery & KubeSpan

Talos nodes publish small, encrypted "affiliate" records (hostname, addresses, role) to a discovery registry; by default that's the hosted service at discovery.talos.dev, which only ever sees encrypted blobs keyed by the cluster ID. That's what talosctl get members shows, and it's how KubePrism learns about every API server. You can self-host the service or disable it. KubeSpan builds on the same data: an optional full-mesh WireGuard network between all nodes, useful when nodes span sites or NATs.

discovery + KubeSpan

cluster:
  discovery:
    enabled: true
    registries:
      service:
        endpoint: https://discovery.talos.dev/   # or your own
      kubernetes:
        disabled: true    # legacy registry using Node annotations
machine:
  network:
    kubespan:
      enabled: true       # WireGuard mesh between all members

$ talosctl -n 10.0.0.10 get members

NODE        NAMESPACE   TYPE     ID         VERSION   HOSTNAME   MACHINE TYPE   OS               ADDRESSES
10.0.0.10   cluster     Member   cp-1       2         cp-1       controlplane   Talos (v1.11.3)  ["10.0.0.10"]
10.0.0.10   cluster     Member   cp-2       2         cp-2       controlplane   Talos (v1.11.3)  ["10.0.0.11"]
10.0.0.10   cluster     Member   cp-3       2         cp-3       controlplane   Talos (v1.11.3)  ["10.0.0.12"]
10.0.0.10   cluster     Member   worker-1   1         worker-1   worker         Talos (v1.11.3)  ["10.0.0.21"]
10.0.0.10   cluster     Member   worker-2   1         worker-2   worker         Talos (v1.11.3)  ["10.0.0.22"]

why it mattersBootstrap is a one-time decision about where etcd is born. Everything else, including other control planes joining, is automatic and repeatable.
why it mattersetcd availability is the control plane's availability: run 3 (or 5) members, replace a failed one before losing another, and take regular etcd snapshots.
why it mattersPick an endpoint that survives a node failure (VIP, LB or DNS), and keep KubePrism on so in-cluster components don't depend on it.