top of page
Search

OpenShift Storage Deep Dive: Volumes, Claims, StorageClasses, CSI and Data Protection

virtualgendham
8 hours ago
15 min read

The architecture article drew storage as one layer in a larger platform. The networking article went deep on the network layer. This one does the same for storage. It follows a write from an application, through a volume mount, a claim, a storage class and a driver, down to the storage backend, and it ties the main claims to Red Hat's OpenShift Container Platform 4.22 documentation. Where the supplied whiteboard simplifies or blurs something, the text says so. The figures in this article are original diagrams drawn from the cited 4.22 documentation. They follow the whiteboard's topics but are not copies of it, and they are not official Red Hat diagrams. The whiteboard itself appears in section 9.

In this article

1. The model in one page

OpenShift storage separates three questions that diagrams often merge: what the application asks for, what the cluster provides, and what the infrastructure actually holds. Three API objects carry those questions.

  • PersistentVolumeClaim (PVC): a request for storage, written by a developer. It asks for a size, an access mode, and usually a storage class. PVCs are namespaced, like pods.

  • PersistentVolume (PV): a piece of storage in the cluster. It is a cluster resource, not scoped to any project, created either by an administrator in advance or dynamically from a StorageClass. A PV bound to a claim cannot be bound to another claim, which effectively scopes it to the claiming namespace (Understanding persistent storage).

  • StorageClass: the menu. It names a provisioner and the parameters that provisioner should use, plus policies such as the reclaim policy and the binding mode. StorageClasses are cluster-wide and only cluster-admin or storage-admin users can create them (Dynamic provisioning).

A pod never touches a PV directly. It references its own namespace's PVC in its volumes block, and the cluster mounts the bound volume into the container at the requested path, such as /data. Everything else in this article hangs off those three objects and the CSI machinery that fulfills them.

One more boundary is worth stating early. Red Hat's documentation is explicit that high availability of the storage itself is left to the underlying storage provider. OpenShift manages the claim, the mount and the lifecycle; the backend decides how redundant the bytes are.

2. Who runs storage: the Operators and CSI

Before looking at volumes, look at who programs them. OpenShift Container Platform 4.22 uses the Container Storage Interface (CSI), version 1.6.0 of the specification, as the mechanism for storage plugins (Using Container Storage Interface). In-tree plugins are being replaced by CSI drivers.


Figure 1. The storage control plane: who provisions, attaches and mounts. Original diagram drawn for this article from the OpenShift 4.22 CSI chapter. Not an official Red Hat diagram.

The pieces, and where they run:

  • Cluster Storage Operator: depending on the platform, it installs a default storage class. That class is owned and controlled by the Operator; it cannot be deleted or modified beyond annotations and labels. If you want different behavior, you define your own storage class (Dynamic provisioning).

  • CSI Driver Operators: for each supported backend, OpenShift installs the driver Operator, the driver itself, and the required storage class by default. One exception matters: the AWS EFS CSI driver is not installed by default and must be installed manually.

  • External CSI controllers: a deployment with five containers - snapshotter, resizer, attacher, provisioner and the driver container. The attacher and provisioner talk to the driver over UNIX domain sockets, so no CSI traffic leaves the pod. These pods handle attach, detach, provision, delete, snapshot and resize calls. Because those calls use storage-backend credentials, Red Hat recommends running the controller pods on infrastructure nodes so credentials never sit on compute nodes.

  • CSI driver daemon set: a pod on every node, containing the driver and a registrar. The registrar registers the driver with the node's openshift-node service over a UNIX socket. The node driver holds as few credentials as possible and implements only node-side calls such as NodeStage and NodePublish. This is the component that actually mounts the volume into the pod.

  • CSI Snapshot Controller Operator: installed by the Cluster Version Operator on every cluster. It runs the snapshot controller in the openshift-cluster-storage-operator namespace and creates the snapshot CRDs: VolumeSnapshot, VolumeSnapshotContent and VolumeSnapshotClass.

oc get clusteroperator storage
oc get pods -n openshift-cluster-storage-operator
oc get sc
oc get csidrivers

The first command is the cluster-wide health check. The second shows the control-plane pods for storage. The last two list the menus and the drivers behind them.

The practical lesson mirrors the networking article's: failure domains. A broken node daemon set pod breaks mounts on that node. A broken external provisioner breaks new claims cluster-wide. Decide which symptom you have before you touch anything.

3. PVs and PVCs: the lifecycle

The PV lifecycle has five phases: provision, bind, use, release and reclaim. Each phase has a behavior that can surprise you in production (Understanding persistent storage).

Provision. Either an administrator creates PVs in advance (static), or a provisioner creates one when a claim appears (dynamic). Static is still common for NFS shares and for storage that exists before the cluster does.

Bind. A control loop watches for new PVCs and matches them to PVs by two criteria only: access mode and size. The binder tries direct matches first, groups volumes with the same modes, sorts them smallest to largest, and binds the smallest volume that satisfies the claim. You can be granted more modes than you asked for, never fewer. A claim that matches nothing stays unbound indefinitely - it binds later, if a matching volume ever appears. With dynamic provisioning, the provisioner creates the volume to order instead.

Use. Pods reference the claim in their volumes block. Storage Object in Use Protection is on by default: deleting a PVC that a pod is using does not remove it until no pod uses it, and deleting a PV that is bound does not remove it until it is unbound. This exists to prevent data loss, and it also explains "stuck terminating" claims.

One operational note from the same chapter: attaching volumes with very high file counts can make pods fail or take a long time to reach Ready, because of the recursive ownership and permission changes at mount time. Red Hat links a Knowledgebase article for mitigation.

Release and reclaim. Deleting the claim releases the volume. What happens next is the reclaim policy's decision:

  • Retain: the PV and the backend asset survive. An administrator reclaims manually: delete the PV object, clean the data on the asset, then either delete the asset or create a new PV from it.

  • Delete: the PV object and the backend asset, such as the EBS volume or vSphere disk, are both removed.

  • Recycle: deprecated in OpenShift 4. Use dynamic provisioning instead.

A PV moves through the phases Available, Bound, Released and Failed. The status.lastPhaseTransitionTime field records each transition, which is useful when you are reconstructing what happened to a volume overnight.

The reclaim policy is not fixed at creation. You can patch it on the PV object, for example from Delete to Retain before decommissioning an application you might need back:oc patch pv <pv_name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

Figure 2. The persistent volume lifecycle and the four volume phases. Original diagram drawn for this article from the OpenShift 4.22 Understanding persistent storage chapter. Not an official Red Hat diagram.

4. StorageClasses and dynamic provisioning

Dynamic provisioning inverts the old flow. Instead of an administrator pre-building a pool of PVs, the administrator writes storage classes, and each claim triggers the backend to create a volume of exactly the requested size and type (Dynamic provisioning).

A StorageClass carries more policy than most diagrams show:

  • provisioner: the plugin name, such as ebs.csi.aws.com or csi.vsphere.vmware.com.

  • parameters: provisioner-specific settings. For AWS EBS this includes the volume type (gp3 by default; io1, sc1, st1 also listed), iopsPerGB for io1 (Red Hat notes the 20,000 IOPS ceiling from AWS), encryption and a KMS key ARN, and the filesystem, ext4 by default.

  • reclaimPolicy: what happens to the volume when its claim is deleted. The whiteboard's summary is right: Delete removes the PV with the claim; Retain keeps it for manual cleanup.

  • volumeBindingMode: Immediate provisions as soon as the claim exists. WaitForFirstConsumer delays provisioning until a pod using the claim is scheduled, so the volume lands in the same zone as the node. Red Hat strongly recommends WaitForFirstConsumer on zoned storage such as Azure Disk and GCE PD.

  • allowVolumeExpansion: whether claims against this class can grow later. See section 7.

  • mountOptions: per-class mount flags, for example uid and gid for Azure File, whose mounted directory owner otherwise differs from the container process UID.

One class per tier is the usual pattern. Red Hat's own worked example uses two classes, gp3 and standard, and switches the default between them by moving the storageclass.kubernetes.io/is-default-class annotation. A PVC that names no class gets the default; a cluster with no default and a classless PVC stays Pending.

The default class belongs to the Operator. The Cluster Storage Operator may install one depending on your platform, and you cannot delete or modify it beyond annotations and labels. To change what "default" means, you annotate your own class rather than editing the Operator's.

Figure 3. Dynamic provisioning, step by step: claim, class, provisioner, backend, bound PV. Original diagram drawn for this article from the OpenShift 4.22 Dynamic provisioning chapter. Not an official Red Hat diagram.

5. Volume types, access modes and volume modes

The whiteboard divides storage into block, file and object. The division is real, but the details carry the design decisions.

Access modes

Access modes describe what a volume can do, per Red Hat's table (Understanding persistent storage):

Mode

Meaning

ReadWriteOnce (RWO)

Mounted read-write by a single node

ReadWriteOncePod (RWOP)

Mounted read-write by a single pod on a single node

ReadOnlyMany (ROX)

Mounted read-only by many nodes

ReadWriteMany (RWX)

Mounted read-write by many nodes

Two corrections to common assumptions, both straight from the documentation. First, access modes are capabilities, not enforced constraints: the storage provider surfaces invalid use as runtime mount errors. Second, iSCSI and Fibre Channel have no fencing mechanism. During a node drain the same volume can end up attached to two nodes; Red Hat's instruction is to delete the pods that use the volume before draining.

RWO has a sharper edge worth knowing before you need it. If a node fails, an RWO volume stays attached to the failed node and the replacement pod cannot mount it - the classic Multi-Attach error. Red Hat's guidance for critical workloads is to force-delete the pod on the shutdown node, and its table notes a recreate deployment strategy for pods that rely on AWS EBS.

Which modes a backend offers is a per-plugin fact, not a guess: EBS and Azure Disk are RWO/RWOP only; EFS, Azure File, GCP Filestore and NFS support all four; vSphere gains RWX only when the underlying environment has the vSAN file service, otherwise an RWX request fails at provisioning time; OpenShift Data Foundation supports RWO and RWX.

Volume modes

Separate from access modes, volumeMode chooses how the volume is presented. Filesystem, the default, formats the volume with a filesystem. Block hands the pod a raw block device for the application to use directly, which is why the whiteboard lists block volumes for workloads such as databases. Pods using raw block volumes must run privileged, and support is per-plugin: EBS, Azure Disk, GCP PD, iSCSI, Fibre Channel, local volumes, LVM Storage, ODF and vSphere support block; NFS, EFS, Azure File and CIFS/SMB do not.

Object storage

Object storage does not fit the PV/PVC shape at all. There is no mount; applications talk S3. On OpenShift this is the Multicloud Object Gateway (MCG), part of OpenShift Data Foundation. The naming is not loose: Red Hat's 4.22 object-storage documentation is titled "using the Multicloud Object Gateway (NooBaa)", so MCG is the product name and NooBaa is the technology the whiteboard references. Its documented uses match the whiteboard's: backups, logs and artifacts. Applications typically get buckets through object bucket claims rather than PVCs.

6. Storage backends you will actually meet

The whiteboard's backend table is a good map. These notes add the 4.22 specifics that change designs (Configuring persistent storage; Using Container Storage Interface; the OpenShift Data Foundation documentation).

Cloud block (AWS EBS, Azure Disk, Google Persistent Disk): RWO, zone-scoped, dynamically provisioned through CSI. Snapshots and resize are supported on all three. Watch zone placement - that is what WaitForFirstConsumer is for. On Azure, Red Hat supports only Managed disks. GCP hyperdisk-balanced adds a shared-disk RWX variant with documented limits, including a maximum of eight instances and no snapshot or clone support in that mode.

Cloud file (AWS EFS, Azure File, GCP Filestore): full access-mode support, the standard answer for shared data across pods. EFS is NFSv4 under the hood, and its CSI driver must be installed manually. Azure File has filesystem limitations - no symlinks, hard links or sparse files by default - and its mount options (uid, gid, mfsymlinks) are the documented workaround.

vSphere CNS: provisioned by csi.vsphere.vmware.com. Block and file both work, but RWX depends on the vSAN file service, and online expansion requires vSphere 8.0 Update 1 or later.

NFS: the simple shared-storage option, and the one the whiteboard oversells. Red Hat's dynamic provisioning table lists no built-in NFS provisioner; NFS PVs are typically static, created by an administrator against an existing export. NFS supports all access modes but no block volumes and no mount options beyond what you set on the PV itself. A community NFS CSI driver exists outside Red Hat's supported table - check the Red Hat ecosystem portal before building on it.

Local storage: the Local Storage Operator and LVM Storage cover node-local disks. Local volumes are RWO and obviously not portable between nodes; they suit workloads that replicate their own data, such as distributed databases, and single-node OpenShift.

Red Hat OpenShift Data Foundation (ODF): the whiteboard's first row, and the only backend here that is a separate product with its own documentation set. ODF runs as an operator and provides Ceph-based block (RBD), file (CephFS) and object (RGW) storage, installed into the cluster itself or consuming an external Ceph cluster. Its default storage classes are ocs-storagecluster-ceph-rbd and ocs-storagecluster-cephfs, with provisioners openshift-storage.rbd.csi.ceph.com and openshift-storage.cephfs.csi.ceph.com - note the openshift-storage prefix that the whiteboard's short forms drop. RBD serves RWO workloads; CephFS serves RWX and shared data. ODF supports snapshots (including volume group snapshots, a Technology Preview), cloning and resize, and it is also the object storage behind OADP backups in many designs.

7. Growing and tuning volumes in place

Two 4.22 mechanisms change a volume after creation without reprovisioning it (Expanding persistent volumes; Understanding persistent storage).

Expansion. Three prerequisites: the CSI driver supports resize, the volume was dynamically provisioned, and the StorageClass has allowVolumeExpansion: true. The operation itself is editing .spec.resources.requests.storage on the PVC. OpenShift adds a Resizing condition to the claim and removes it when the expansion completes. Shrinking is not supported, full stop - so the planning question is never "can we shrink later" but "what growth path do we want."

Volume Attributes Classes. Newer and easy to miss: a VolumeAttributesClass lets an administrator define performance parameters - IOPS and throughput - that a PVC can adopt without reprovisioning, by setting volumeAttributesClassName. Support is currently limited to AWS EBS and GCP PD (hyperdisk-balanced only on GCP). Three catches from the documentation: the class's parameters are immutable once defined; you can swap a PVC to a different class but cannot remove the class reference without recreating the PVC; and a provider can take hours to reconcile rapid repeated changes. A metric, openshift_cluster_storage_vac_mismatch_parameters, flags classes on the same driver that define different parameter sets, because unset parameters keep their previous values rather than resetting to defaults.

8. Data protection: snapshots, clones, backups and replication

The whiteboard's four data-protection items map cleanly onto real features, with one naming update (Using Container Storage Interface; Backup and restore; the ODF disaster recovery documentation).

Snapshots. CSI volume snapshots are supported by default with drivers that implement them - check the per-driver table before promising anyone an RPO. The object model deliberately mirrors PV/PVC: a VolumeSnapshot is a namespaced request, a VolumeSnapshotContent is the cluster-scoped real snapshot, and a VolumeSnapshotClass carries the backend parameters. The CSI Snapshot Controller Operator supplies the controller; the vendor's csi-external-snapshotter sidecar triggers the actual CreateSnapshot and DeleteSnapshot calls. Restoring means creating a new PVC whose dataSource names the snapshot. In-tree and FlexVolume drivers cannot do snapshots at all.

Cloning. A new PVC can name an existing PVC as its dataSource, producing a copy - the whiteboard's "create a new PVC from a snapshot" is one path; cloning straight from a PVC is the other. Support is per-driver, as the feature table shows.

Backups. Snapshots are not backups: they live on the same storage as the source. Application backup on OpenShift is OADP, the OpenShift API for Data Protection, which uses Velero. It backs up Kubernetes resources and internal images at namespace granularity, and PVs either through snapshots or through file-system backup (Kopia or Restic). OADP requires object storage for the backup target - S3-compatible, which includes ODF - and offers pre/post hooks for application-consistent backups.

Replication and DR. Asynchronous and synchronous replication for disaster recovery is an ODF feature set, not base OpenShift: ODF documents Metro-DR and Regional-DR solutions in their own guide. If DR is on your requirements list, it is an ODF architecture decision, made at design time.


Figure 4. Four data-protection tools, and the different question each one answers. Original diagram drawn for this article from the OpenShift 4.22 and OpenShift Data Foundation 4.22 documentation. Not an official Red Hat diagram.

9. Whiteboard claims, checked

These are the specific statements from the supplied diagram that needed care, with the Red Hat evidence for each. The diagram is a good learning aid. This table is about precision.


The supplied whiteboard sketch that started this article. The table below checks its claims one by one.

Whiteboard claim

Verdict against Red Hat 4.22 docs

PVC is a request for storage; PV is provisioned by an admin or dynamically

Correct. PVCs are namespaced; PVs are cluster resources.

StorageClass defines type, parameters and provisioner

Correct, and it also carries reclaimPolicy, volumeBindingMode, allowVolumeExpansion and mountOptions. Cluster-wide, admin-only.

Storage classes gp3 / efs / ocs-storagecluster-ceph-rbd / ocs-storagecluster-cephfs

Mostly correct names. gp3 matches Red Hat's own example classes. ODF's classes are real, but their provisioners are openshift-storage.rbd.csi.ceph.com and openshift-storage.cephfs.csi.ceph.com, not the short forms shown. EFS needs a manual driver install.

Reclaim: Delete removes the PV with the PVC; Retain keeps it for manual cleanup

Correct. Retain leaves both the PV and the backend asset for an administrator to reclaim by hand.

Block volumes are RWO; file volumes are RWX

Directionally right for cloud block and file. Nuance: iSCSI and Fibre Channel raw block can do RWX; vSphere file RWX needs the vSAN file service; and access modes are capabilities, not enforced constraints.

Object storage via ODF / NooBaa for backups, logs, artifacts

Right uses. Red Hat's product name is the Multicloud Object Gateway; its 4.22 documentation titles the object chapters "Multicloud Object Gateway (NooBaa)". Object storage is consumed as S3 buckets, not through PV/PVC mounts.

Dynamic provisioning: PVC to StorageClass to CSI provisioner to backend to bound PV

Correct flow. Add volumeBindingMode: WaitForFirstConsumer keeps the volume in the pod's zone; Immediate does not wait.

Backend table: ODF, EBS, NFS, vSphere, Azure, GCP

Fair map. Two corrections: NFS has no built-in dynamic provisioner in the 4.22 table, and its "NFS CSI driver" is a community driver outside Red Hat's supported list. ODF is a separate product, installed as an operator.

Access modes RWO, ROX, RWX

Correct, with a fourth documented mode: RWOP (single pod).

Volume modes: Filesystem default, Block raw

Correct. Block-mode pods must run privileged; NFS, EFS, Azure File and CIFS do not support block volumes.

Data protection: snapshots, cloning, OADP (Velero) backups, ODF replication for DR

All real. Snapshots and clones are CSI features with per-driver support; OADP backs up PVs via snapshots or file-system backup to object storage; DR replication is an ODF feature with Metro-DR and Regional-DR guides.

Key operators: CSI drivers, storage operators, ODF operator, OADP operator

Correct set. Add the CSI Snapshot Controller Operator, installed by default, for the snapshot CRDs.

10. A troubleshooting order that follows the layers

Storage failures also layer. Walk them in the same order as the article:

  • Claim: is the PVC Bound? oc describe pvc shows the events. Pending usually means no matching PV and a provisioner that said no - wrong class name, missing default class, exhausted backend, or bad parameters.

  • Provisioner: is the external CSI controller pod healthy? Cluster-wide provisioning failures live there, and its logs name the backend error.

  • Attach and mount: is the volume stuck attaching? Multi-Attach after a node failure is the RWO signature; force-delete the pod on the dead node. Mount errors at pod start point at the node daemon set pod on that node.

  • Mode: did an RWX claim fail on vSphere or a block request land on NFS? Check the backend's real capabilities against the access-mode and block tables.

  • Growth: is a resize stuck? Check driver support, allowVolumeExpansion on the class, and the Resizing condition on the claim.

  • Protection: is a snapshot never readyToUse? The driver may not implement snapshots, or the VolumeSnapshotClass parameters are wrong for the backend.

oc get pvc,pv
oc describe pvc <claim>
oc get sc <class> -o yaml
oc get volumesnapshot,volumesnapshotcontent
oc get pods -n openshift-cluster-storage-operator
Operator status first, here too. A degraded storage Cluster Operator is the first stop when symptoms are cluster-wide; a single failing mount is almost always node-local.

11. Planning checklist

  • Choose backends per workload shape: block RWO for databases, file RWX for shared content, object for backups, logs and artifacts. Write the choice down per application.

  • One StorageClass per tier, with deliberate parameters - type, encryption, filesystem. Set the default class by annotation on purpose; the Operator's default is not your design.

  • Use WaitForFirstConsumer on zoned cloud storage.

  • Decide reclaim policies before data exists: Delete for disposable tiers, Retain for anything you would mourn, and patch PVs when an application's risk changes.

  • Set allowVolumeExpansion: true from day one. There is no shrink.

  • Check the per-driver feature table before promising snapshots, clones or resize to a workload owner.

  • Plan the RWO failure path: recreate strategy where required, and a documented force-delete procedure for Multi-Attach after node loss.

  • If you need EFS, schedule the manual driver install. If you need NFS, plan static PVs or evaluate the community CSI driver deliberately.

  • Treat DR as an ODF architecture decision (Metro-DR or Regional-DR), not a checkbox.

  • Remember that storage HA belongs to the backend. OpenShift manages claims and mounts; redundancy comes from the provider.

  • State the release you design against. This article is based on OpenShift Container Platform 4.22 documentation.

12. Where this fits in the series

The architecture article introduced storage next to networking, security and operations. The networking article expanded the network layer; this one expanded storage. The same method applies to the rest: draw the layers, name the Operator that owns each, and verify each claim in the documentation for your version.

References

The diagrams in this article were drawn for it from the cited documentation. They are not official Red Hat diagrams. Based on OpenShift Container Platform 4.22 documentation, checked November 12, 2026.

Comments

©2026 by GendhamCloud.

bottom of page