PV, PVC & CSI

A PersistentVolumeClaim is a request ("10Gi, one writer"), a PersistentVolume is the volume that satisfies it, and a StorageClass says which driver creates volumes on demand. Since the in-tree volume plugins were removed, that driver is almost always a CSI driver: a set of containers that turn Kubernetes objects into gRPC calls like CreateVolume and NodePublishVolume. Once you know which component makes each call, a PVC stuck in Pending stops being a mystery.

demo 1 · lifecycle

From claim to /data, one call at a time

Step through what happens between kubectl apply and your container seeing a mounted directory. Toggle the volumeBindingMode. With Immediate the disk is created before anyone knows where the pod will run. With WaitForFirstConsumer the scheduler picks the node first and the disk is created in that node's topology. Turn on the nodeSelector to see Immediate fail.

pod has nodeSelector: zone-c

Cluster

step 0/12

control plane

kubectl

you

kube-scheduler

VolumeBinding plugin

kube-apiserver / etcd

PVC data—

PV pvc-7f3a…—

Pod app—

VolumeAttachment—

kube-controller-manager

PV binder · attach/detach

csi controller · Deployment (1 replica, leader-elected)

external-provisioner

watches PVCs

external-attacher

watches VolumeAttachments

external-resizer

idle here

driver (controller)

talks to the storage API

worker-1zone-a

kubelet

csi node plugin

DaemonSet pod

…/globalmount

…/pods/<uid>/…/mount

zone-a storage

worker-2zone-b

kubelet

csi node plugin

DaemonSet pod

…/globalmount

…/pods/<uid>/…/mount

zone-b storage

worker-3zone-c

kubelet

csi node plugin

DaemonSet pod

…/globalmount

…/pods/<uid>/…/mount

zone-c storage

What just happened

Press Play or Step. Then flip the binding mode and the nodeSelector toggle to see what changes.

CSI calls so far

none yet

storageclass + pvc + pod

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast
provisioner: csi.proxmox.sinextra.dev   # = CSIDriver name; picks the driver
volumeBindingMode: WaitForFirstConsumer  # let the scheduler choose the node first
reclaimPolicy: Delete                    # or Retain
allowVolumeExpansion: true
parameters:                              # opaque to Kubernetes, passed to CreateVolume
  storage: local-lvm
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
spec:
  storageClassName: fast
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: 10Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: app
      image: busybox
      command: ["sh", "-c", "echo hi > /data/hello && sleep infinity"]
      volumeMounts:
        - { name: data, mountPath: /data }
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: data

debugging a stuck volume

kubectl get pvc data -o wide                 # Pending? read the events:
kubectl describe pvc data                    # "waiting for first consumer", provisioning errors
kubectl get pv                               # the PV + its claim, reclaim policy, status
kubectl get pv <pv> -o jsonpath='{.spec.nodeAffinity}'   # where the volume can be used
kubectl get volumeattachments                # attach status per node
kubectl get csidrivers,csinodes              # which drivers are registered where
kubectl -n <csi-ns> logs deploy/<controller> -c csi-provisioner   # sidecar logs

Why WaitForFirstConsumer is the right default

Any volume with topology, such as a disk on a specific Proxmox host, a node-local path, or a zonal cloud disk, should use WaitForFirstConsumer. Otherwise the volume's location silently overrides your pod's scheduling constraints. Immediate is only harmless for storage reachable from every node equally (Ceph RBD, NFS).

demo 2 · components

Who is who in a CSI driver

CSI deliberately keeps the vendor's code free of Kubernetes. The driver only speaks gRPC on a unix socket. Kubernetes-aware sidecars maintained by sig-storage watch the API and translate. A driver ships as a controller Deployment (cluster-wide operations: create, attach, resize, snapshot) and a node DaemonSet (format and mount on each node), plus a CSIDriver object.

Anatomy of a CSI driver · click anything

API objects

controller · Deployment1–2 replicas, leader election

shared emptyDir · /csi/csi.sock (gRPC)

node · DaemonSet (every node)privileged

/var/lib/kubelet/plugins/<driver>/csi.sock

/var/lib/kubelet/plugins_registry/ ← registration

/var/lib/kubelet/plugins/kubernetes.io/csi/… ← staging

/var/lib/kubelet/pods/<uid>/volumes/… ← publish

/dev ← block devices

external-provisioner

sidecar in the controller Deployment

watches

PersistentVolumeClaims whose StorageClass.provisioner matches the driver

CSI

CreateVolume / DeleteVolume

Turns a claim into a real volume and a PV. Honors volumeBindingMode: with WaitForFirstConsumer it waits for the scheduler's selected-node annotation and forwards that node's topology. On PVC deletion with reclaimPolicy Delete, it calls DeleteVolume and removes the PV.

demo 3 · access modes

Access modes: the "once" is per node

Access modes describe how many nodes (or with RWOP, pods) may mount a volume at once, and they only mean something if the driver supports them. Switch the backend between a block disk and a shared filesystem.

Access modes · 4 pods share one PVC

Read-write by a single node, not a single pod. Every pod on the node holding the attachment can mount it. A pod on another node gets stuck in ContainerCreating with a Multi-Attach error until the first node lets go.

worker-1

worker-2

worker-3

Hover or tap a pod to see why.

demo 4 · reclaim

Reclaim policy: who owns the data after the claim is gone?

The StorageClass's reclaimPolicy is copied onto every PV it provisions. Changing the class later doesn't touch existing PVs. Patch persistentVolumeReclaimPolicy on the PV itself to protect something important.

Reclaim policy · what deleting a PVC does

PVC data

Bound

→

PV pvc-7f3a…

Bound

→

backend disk

exists · 10Gi of data

PV persistentVolumeReclaimPolicy: Delete (copied from the StorageClass's reclaimPolicy at provisioning time). Delete the claim to see what happens.

expansion · snapshots · statefulsets

Day 2: grow, snapshot, replicate

Volumes grow but never shrink. Snapshots are an add-on API: you need the snapshot.storage.k8s.io/v1 CRDs and the snapshot-controller installed, plus a driver whose controller pod runs the csi-snapshotter sidecar. A StatefulSet's volumeClaimTemplates give each replica its own claim, and that claim follows the replica's stable name across restarts and reschedules.

online volume expansion

# StorageClass must have allowVolumeExpansion: true
kubectl patch pvc data -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'

# external-resizer -> ControllerExpandVolume (grow the backend disk)
# kubelet         -> NodeExpandVolume       (grow ext4/xfs, usually online)
kubectl get pvc data -o jsonpath='{.status.capacity.storage}'   # 20Gi when done

statefulset with per-replica claims

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: pg
spec:
  serviceName: pg
  replicas: 3
  persistentVolumeClaimRetentionPolicy:   # default: Retain / Retain
    whenDeleted: Retain                   # keep PVCs if the StatefulSet is deleted
    whenScaled: Delete                    # drop pg-data-pg-2 when scaling 3 -> 2
  selector: { matchLabels: { app: pg } }
  template:
    metadata: { labels: { app: pg } }
    spec:
      containers:
        - name: pg
          image: postgres:17
          volumeMounts:
            - { name: pg-data, mountPath: /var/lib/postgresql/data }
  volumeClaimTemplates:                   # one PVC per replica: pg-data-pg-0, -1, -2
    - metadata: { name: pg-data }
      spec:
        storageClassName: fast
        accessModes: [ReadWriteOncePod]
        resources: { requests: { storage: 20Gi } }

snapshot + restore

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: csi-snap
driver: rook-ceph.rbd.csi.ceph.com   # must match the PV's driver
deletionPolicy: Delete
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: data-before-upgrade
spec:
  volumeSnapshotClassName: csi-snap
  source:
    persistentVolumeClaimName: data
---
# restore = a new PVC whose dataSource is the snapshot
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-restored
spec:
  storageClassName: ceph-block
  dataSource:
    apiGroup: snapshot.storage.k8s.io
    kind: VolumeSnapshot
    name: data-before-upgrade
  accessModes: [ReadWriteOnce]
  resources: { requests: { storage: 10Gi } }

talos specifics

On Talos: where do bytes actually live?

Talos has no writable root filesystem. Everything stateful lives on the EPHEMERAL partition mounted at /var, where etcd, container images, kubelet state and any hostPath data sit side by side. It survives reboots but is wiped by talosctl reset, so it's node-scoped state, not durable storage.

Three things that bite on Talos

kubelet runs in a container. Host paths that hostPath-style provisioners use must also be mounted into kubelet via machine.kubelet.extraMounts, with rshared propagation. Otherwise the pod gets a directory that doesn't match what the provisioner created.
Filling /var hurts the node. A runaway local-path volume competes with etcd and image storage on the same partition, and kubelet starts evicting pods under disk pressure. With Talos 1.10+ you can carve a separate partition with a UserVolumeConfig (mounted under /var/mnt/<name>, which Talos already mounts into kubelet, so no extraMounts needed for local-path) and point local storage there. You can also cap EPHEMERAL's size with a VolumeConfig.
Host tools come from extensions. Drivers that shell out to iSCSI or block tools on the host need the matching Talos system extensions (e.g. siderolabs/iscsi-tools, siderolabs/util-linux-tools) baked into the image from Image Factory. See the StorageClasses page.

machine config patch · kubelet extraMounts

# Talos machine config patch: let kubelet (a container) see a host path
# used by a hostPath-based provisioner such as local-path-provisioner
machine:
  kubelet:
    extraMounts:
      - destination: /var/local-path-provisioner
        type: bind
        source: /var/local-path-provisioner
        options: [bind, rshared, rw]

machine config document · UserVolumeConfig

# Talos 1.10+: carve a dedicated partition for workloads instead of sharing /var
apiVersion: v1alpha1
kind: UserVolumeConfig
name: local-storage            # mounted at /var/mnt/local-storage
provisioning:
  diskSelector:
    match: disk.transport == "nvme" && !system_disk
  minSize: 100GB
  maxSize: 500GB
filesystem:
  type: xfs

inspect volumes on a node

talosctl -n 10.0.0.21 get discoveredvolumes   # disks/partitions Talos sees
talosctl -n 10.0.0.21 get volumestatus        # EPHEMERAL, user volumes, their phase
talosctl -n 10.0.0.21 get mountstatus         # what is mounted where
talosctl -n 10.0.0.21 ls /var/mnt/local-storage

why it mattersA PVC is only a request. The external-provisioner sidecar turns it into a real volume plus a PV, and kubelet (through the node plugin) turns that into a mount. When something is stuck, find which sidecar should be acting and read its logs and the PVC events.
why it mattersUse WaitForFirstConsumer for anything with topology, which on Proxmox and bare metal is almost everything. Otherwise the disk is placed before the scheduler has a say, and pods end up unschedulable with "volume node affinity conflict".
why it mattersRWO means one node, not one pod. Use RWOP when you need a single writer. Choose Retain for data you can't recreate, because the default Delete removes the backend disk the moment the claim goes away.