Kubernetes Storage: PVCs, StorageClasses, and the Pod That Stays Pending

The deploy went out and the pod never started. kubectl get pods showed Pending, and kubectl describe pod ended with an event about a persistent volume claim that nobody had created by hand. There was memory on the node. There was CPU. Storage had not come up in conversation since the cluster was built.

This is where most Kubernetes storage trouble lives. Not in disks failing, but in the gap between what a deployment asks for and what the cluster is willing to bind to it. The API accepts the request, the scheduler has no volume to place, and the pod sits there reporting nothing more dramatic than unschedulable.

The three objects and what each is for

• PersistentVolume (PV): a piece of storage that exists on its own, either created by hand or by a provisioner.
• PersistentVolumeClaim (PVC): a request for one. A pod never names a disk; it names a claim.
• StorageClass: the rule for satisfying claims that have no volume yet, including the provisioner, its parameters, and what happens to the volume when the claim goes away.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-data
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: gp3
  resources:
    requests:
      storage: 10Gi

The indirection is the point: a deployment asks for 10Gi and lets this cluster decide what 10Gi means here.

Why binding fails quietly

Four situations account for almost everything I have debugged.

No provisioner. The class names a plugin the cluster does not have. On EKS that usually means the EBS CSI driver was never installed. On a bare metal cluster it means nothing is answering at all. The claim stays Pending with no event that reads like an error.

Access mode. ReadWriteMany is not offered by most block storage. A claim asking for it against a gp3 class never binds, and the pod waiting on it never starts.

Zone. A volume in one availability zone cannot attach to a node in another. The scheduler knows this and declines the pod instead of attempting the attach.

Size. A class with a minimum larger than your request will not satisfy it, and the claim waits.

kubectl get pvc
kubectl describe pvc app-data
kubectl get sc

Read the events on the claim, not the pod. The pod only reports that nobody handed it a volume. Nothing has failed yet.

StatefulSets, and why the volume follows the identity

A Deployment with one replica and one claim works right up until you scale it. ReadWriteOnce allows a single node at a time. A second replica landing elsewhere never mounts, and the pod hangs in Pending.

StatefulSets exist partly for this. Each pod gets a stable identity and its own claim, so the volume for replica zero belongs to replica zero and survives that pod being deleted and recreated.

volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      storageClassName: gp3
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 20Gi

Note what is not in that snippet: a claim retention policy. Without one, deleting the StatefulSet leaves the claims behind. That is usually what you want during an incident, and rarely what you want in a CI namespace that gets wiped nightly.

The check order I actually use

Order matters, because the cheap checks rule out the expensive ones.

1. kubectl get pvc: is the claim Bound?
2. If not, describe the claim and read its events.
3. kubectl get sc: does the class exist, and is one marked default?
4. Is the node in the same zone the volume landed in?
5. Does the access mode match what this storage offers?

That sequence, inside the wider pod triage order, resolves nearly every storage-related Pending I have seen.

Deleting on purpose

reclaimPolicy decides what happens to the underlying volume when its claim is removed: Retain leaves it, Delete destroys it. Most managed classes default to Delete, which is right for stateless environments and is also how people lose a database during a careless namespace cleanup.

For anything holding real data I set Retain and remove the volume as its own audited step, the same way restore drills treat deletion as something you choose rather than inherit.

Summary

Pods ask for claims, claims bind through classes, and the class needs a provisioner this cluster actually has. When a pod sits Pending with a storage event, read the claim's events first, then check the access mode and the zone. Use a StatefulSet with volume claim templates when each replica needs its own disk, set the reclaim policy deliberately, and treat Delete as a decision you make rather than one you inherit.