Storage providers
SHC supports three storage provider types for stack volumes and backups:
| Provider | Typical use | Shared across nodes | Snapshots | Quotas |
|---|---|---|---|---|
local | Single-node, or per-node local fast storage | No | ZFS/Btrfs only | ZFS/XFS/Btrfs only |
nfs | Multi-node shared storage, simple setup | Yes | rsync-based | No |
cephfs | Multi-node with native snapshotting | Yes | Native | Limited |
An SHC cluster can have multiple providers, e.g. local for
per-node fast volumes plus nfs for shared backups. Each stack
picks which provider to use based on the app’s requires.storage
whitelist and operator-chosen preferences.
Source of truth (ce module):
modules/storage/storageapi/types.go
(types, ProviderConfig, the Capabilities set),
modules/storage/providers/
(the provider implementations),
modules/storage/commands/
(CLI).
For the per-stack resolution logic (how SHC picks which provider
for a given stack) see
architecture/modules/storage.md.
Cloud block volumes are a fourth thing, and they have their own rule
Section titled “Cloud block volumes are a fourth thing, and they have their own rule”The three providers above are where a stack’s volumes and backups are served from. Separately, an estate whose cloud sells attachable disks (Hetzner, AWS EBS, Linode, Vultr, Azure, Exoscale) can put a deployment’s durable mounts on real block devices, and on such an estate that is the default rather than an opt-in.
Which mounts are durable is not a new judgment: backup.yaml already carries
it. See Block storage by default for the
decision procedure, how to declare a volume ephemeral, why object storage and
block storage answer different questions, and why an existing deployment is
flagged rather than switched.
The default: local
Section titled “The default: local”Every SHC installation starts with a local provider pointing at
the daemon’s state directory (typically /var/lib/shc/storage). No
configuration needed; shc init sets it up.
Capabilities are auto-detected by
detection.go based on
the filesystem backing the state directory:
- ZFS →
supports_quotas=true,supports_snapshots=true, native snapshots, native quotas - XFS (with project quotas enabled) →
supports_quotas=true, snapshot via rsync - Btrfs →
supports_snapshots=true(CoW snapshots),supports_quotas=true(qgroups) - ext4 →
supports_quotas=false,supports_snapshots=false(rsync fallback for snapshots works, but expensive)
To see what was detected:
shc storage lsshc storage show localPicking a good filesystem for local
Section titled “Picking a good filesystem for local”- Production single-node, heavy IO: ZFS with a dedicated dataset. Best snapshot + quota + IO story SHC supports.
- Production multi-node with local-mode stacks: Btrfs subvolumes, or ZFS if you have it. Fall through to NFS / CephFS for shared volumes.
- Development or small deployments: ext4 is fine. You won’t get instant snapshots but rsync-based backup will still work, just slower.
Adding an additional local provider
Section titled “Adding an additional local provider”You can add a second local provider pointing at a different mount, e.g. a fast SSD for DB data, keeping the default provider on slower storage:
shc storage add fast-nvme --path /mnt/nvme/shc --node node-ashc storage link fast-nvme node-a node-b # attach to specific nodesOptions:
--path: mount path. Must be writable by theshcuser.--node: single-node scope. Omit for cluster-wide.
A node is named as shc node ls shows it, or by its id. shc storage show
and shc storage ls print the linked nodes by name.
Commands:
shc storage add <name> --path <path> [--node <node>]:commands/add.goshc storage link <name> <node> [<node>...]: attach a provider to specific nodes (commands/link.go)shc storage unlink <name> <node>: detachshc storage rm <name>: remove a provider (refuses if stacks are using it)
Reserved names that you cannot use for a user-added provider:
local, nfs, cephfs: these are the managed singletons created
via their dedicated enable commands.
NFS: simple shared storage
Section titled “NFS: simple shared storage”When to use NFS
Section titled “When to use NFS”- Multi-node cluster where the same stack can schedule to any node and needs the same data
- Shared backup repository (
backup.repoindefault.yaml) - You want the simplest possible shared-storage setup: NFS “just works” over a LAN
What SHC provisions for you
Section titled “What SHC provisions for you”SHC ships a managed NFS singleton: one NFS server across the cluster, with SHC handling exports, client mounts, and reconnects.
shc storage nfs enable --export-path /mnt/shc/nfs --network 10.0.0.0/24What happens:
- SHC designates one node as the NFS server (the one you ran the
command on, unless overridden) and installs
nfs-kernel-server. - Exports
<export-path>to the given network CIDR. - Installs
nfs-commonon every other cluster node. - Mounts the export on each client node under
/mnt/shc/nfs-mount(or similar; seeshc storage show nfs). - Registers an
nfsprovider in the control-plane database pointing at the mount.
Source:
modules/storage/commands/managed.go (the shc storage nfs verbs),
modules/storage/shared_manager.go + modules/storage/nfs/ (exports,
cleanup).
NFS capabilities & caveats
Section titled “NFS capabilities & caveats”- Shared access: yes. That’s the whole point.
- Snapshots: rsync-based. SHC stages snapshots by rsync’ing to
a
.snapshotsdirectory on the NFS export. Works on any backing filesystem but isn’t atomic. - Quotas: no. NFS doesn’t model per-directory quotas in a way SHC can enforce.
- Locking: SHC uses application-level locking (database-backed) for concurrency, so NFS’s own locking quirks don’t bite you.
- Performance: fine for most web-app workloads. Not suitable
for databases that fsync heavily; run those on
localstorage and only back them up to NFS.
Operations
Section titled “Operations”shc storage nfs enable # create the singleton (run once per cluster)shc storage nfs status # check server & client mount healthshc storage nfs join -n home02 # pin home02 as the NFS server, this node without -nshc storage nfs leave # hand the NFS server role from this node to anothershc storage nfs disable # tear down the singleton (CAREFUL: affects all nodes)join moves the server role, not the data. The new server stops mounting
the share and serves its own /mnt/shc/nfs; the files stay on the old
host. A share mount the new server still has in use refuses the join
until whatever holds it open is stopped.
Add a third-party / external NFS
Section titled “Add a third-party / external NFS”If you already have an NFS server you want SHC to consume (instead of letting SHC manage one), add it as an external provider:
# Mount the NFS export yourself (via /etc/fstab or autofs),# then register it as a local-type external provider:shc storage add team-nfs --path /mnt/team-nfsSHC will treat it like a local provider backed by whatever
filesystem the mount exposes.
CephFS: shared storage with native snapshots
Section titled “CephFS: shared storage with native snapshots”When to use CephFS
Section titled “When to use CephFS”- You already run Ceph for other workloads and want SHC to participate
- You need native atomic snapshots of shared storage (NFS snapshots aren’t atomic)
- Larger clusters where you want to decouple storage from SHC-managed infrastructure
Enabling
Section titled “Enabling”CephFS is configured by connecting SHC to an existing Ceph cluster; SHC does not provision Ceph for you.
shc storage cephfs enable [--cluster-network CIDR] [--public-network CIDR]--cluster-network and --public-network are optional. Omit both when Ceph should use the host’s default networks.
SHC will:
- Validate the Ceph credentials can reach a monitor.
- Create the SHC subdirectory on the CephFS filesystem if needed.
- Mount it on every cluster node via
mount -t ceph .... - Register a
cephfsprovider in the control-plane database.
Source:
modules/storage/commands/managed.go (the shc storage cephfs verbs),
modules/storage/ceph/manager.go.
CephFS capabilities & caveats
Section titled “CephFS capabilities & caveats”- Shared access: yes
- Snapshots: native, atomic. Faster and safer than NFS+rsync
- Quotas: limited. Ceph supports directory quotas; SHC exposes them where the subdirectory is set up with quota attributes
- Setup complexity: you need a working Ceph cluster first. If you don’t already run Ceph, NFS is the pragmatic choice.
Cloud block volumes: real disks per mount
Section titled “Cloud block volumes: real disks per mount”Beyond the filesystem providers above, SHC can back an individual compose mount with a provider-managed block volume: a real cloud disk (Hetzner Cloud volume, OpenStack Cinder, AWS EBS, GCE PD, Azure Disk, DigitalOcean, Linode, Vultr, CloudStack, UpCloud, Oracle Cloud, Alibaba Cloud) that is created, attached, and mounted for that one mount, instead of a directory on the node’s filesystem.
An app opts a mount in with the x-shc-volumes compose extension:
x-shc-volumes: pgdata: size: 10Gi # Ki/Mi/Gi/Ti (binary), K/M/G/T (decimal), or bare bytes access: rwo # rwo = single-node read-write (default), rox = read-only multi-attachvolumes: pgdata:What you get compared to a plain directory:
- Survives the node. The disk is a first-class cloud resource; a deployment move re-attaches it to the target node (same location) instead of copying files.
- Sized and enforced per mount:
sizeis the disk’s real capacity, not a quota approximation. - Backup-reconstructable. Backups record the volume identity, and a restore can re-provision + re-attach the disk from the backup when the original is gone.
- Leased single-writer.
rwovolumes carry a monotonic attach lease, so a partitioned node can never keep writing after the cluster rebinds the volume elsewhere.
rwo and rox are the only accepted access modes. An unsupported value
returns X400328VOLACC at install. Read-write-many data belongs on a shared
filesystem such as CephFS or NFS.
The cloud credentials come from the connection registry (shc connection), the same place the cluster’s provisioning credentials
live: the storage module resolves the node’s provider (for example
hcloud on a Hetzner node) and calls the cloud API with the
registered connection. Nodes without a block-volume provider fall back
to the app’s whitelisted filesystem providers.
Looking inside a block volume
Section titled “Looking inside a block volume”shc volume mount <id> binds a volume’s data read-only under the daemon’s
state directory. A block volume’s data is the filesystem on its device, and
the directory its deployment mounts it at is an empty directory on the node’s
own disk whenever the volume is not mounted there, so the daemon binds that
directory only while the volume is mounted at it:
- Attached to the daemon’s node and mounted there, it is bound.
- Attached to the daemon’s node but not mounted, say on a node back from a
power cut before its remount, it is refused with
X409886VLMNMT. - Attached to another node, it is refused with
X409896VLMHLD. Run the mount on that node. - Detached, as an uninstall that kept it or a failed install’s rollback leaves
it, it is attached to the daemon’s node through the same steps an install
takes, its device mounted, then bound. The output says it attached the
volume. Releasing the mount detaches it again, whether by
shc volume umount, the lease running out, or the reap after a restart. A device another node holds open is refused withX409896VLMHLDbefore anything changes, and an attach that fails is undone and refused withX500897VLMATT.
Volume providers and the nodes they reach
Section titled “Volume providers and the nodes they reach”A block volume reaches only the nodes whose recorded cloud and location match the volume’s. Each volume provider belongs to one cloud:
| Volume provider | Node cloud |
|---|---|
hcloud | hetzner |
aws-ebs-volume | aws |
gcepd | gcp |
azure-disk | azure |
dovolume | digitalocean |
vultr-volume | vultr |
linode-volume | linode |
cinder-volume | openstack |
cloudstack-volume | cloudstack |
incus-volume | incus |
upcloud-storage | upcloud |
oci-volume | oci |
alibaba-disk | alibaba |
linstor | any: every node inside the connection’s location |
The location is the place the provider creates the volume in, and a node
is inside it when its location is that place or a path under it, read
either as the node’s own --location or as the path shc node ls prints,
under its cloud: a node at --cloud hetzner --location fsn1 is inside
fsn1, hetzner/fsn1 and hetzner. A volume in use also reaches the node
it is attached on, whatever its location says. For
upcloud-storage it is the connection’s zone (fi-hel1), for
oci-volume the availability domain’s label in lower case (phx-ad-1),
for alibaba-disk the connection’s zone id (cn-hangzhou-h).
A node reads its cloud and location from the cloud’s metadata service on
Hetzner, AWS, GCP, Azure, UpCloud and DigitalOcean. On any other cloud,
Oracle Cloud and Alibaba Cloud among them, say both at shc init or
shc node join, or later with shc node update:
$ shc node update <node> --cloud oci --location phx-ad-1shc node ls shows what each node records. A node with no cloud, or with
a location outside the volume’s, is outside the volume, and a deployment
whose volume reaches no node stays pending. The per-cloud pages have the
exact values: UpCloud,
Oracle Cloud, Alibaba Cloud.
LINSTOR: volumes on the nodes’ own disks, copied across hosts
Section titled “LINSTOR: volumes on the nodes’ own disks, copied across hosts”When the SHC nodes are themselves LINSTOR hosts, as on an Incus cluster where
shcd runs on each host, a linstor connection puts a mount on a DRBD resource
with copies on several hosts:
$ shc connection connect --type linstor --name ogden-linstor \ --endpoint http://10.22.0.11:3370,http://10.22.0.12:3370,http://10.22.0.13:3370 \ --cred resource_group=shc-volumes --cred location=ogdenendpoint lists the controllers. LINSTOR runs one at a time and the one that
answers is used. resource_group is the group a volume is spawned from, and
its place count is how many hosts hold a copy. Connect refuses a group the
controller does not have (X400291LNSCFG). location is the location path the
hosts sit under. A volume reaches every node inside it, whatever cloud the node
records. Connect takes a location no node is in yet, for nodes that join later,
and storage.connection-reach warns while it holds none. ca_cert and
tls_server_name pin an https controller.
A mount is attached on the node that runs its replica: a node with a copy uses it, and any other node gets a diskless resource that reads and writes the copies over the network. DRBD lets one node at a time open the device, and only while that node keeps quorum. A move, or a move off a lost host, attaches the same resource on the target and copies nothing then.
A diskless node pays the network on every read and waits on two peers for
every write, so SHC sets LINSTOR’s DrbdOptions/auto-diskful 1 and
DrbdOptions/auto-diskful-allow-cleanup true on each volume it provisions or
attaches. Once a diskless resource has been primary for a minute, LINSTOR
makes a copy on its node, and once that copy is UpToDate it deletes a spare
one, so the volume holds the group’s place count of copies before, during and
after. After a move the new node gets the copy the same way. A key set on the
resource definition, the resource group or the controller is left as the
operator set it. The new node needs a storage pool of the resource group to
hold a copy.
DRBD promotes the node that opens the device for writing, inside the open,
and every peer must agree. A peer refuses while it holds the device open
read-only, and an LVM scan (lvs, lvdisplay, Incus’s LVM driver) opens
every block device it is not told to skip. On every LINSTOR host, keep LVM off
the DRBD devices in /etc/lvm/lvm.conf:
devices { global_filter = [ "r|^/dev/drbd|" ] }lvmconfig devices/global_filter shows it, and shc doctor run --category storage fails a host with drbd loaded whose filter lets LVM open
/dev/drbd* (storage.lvm-drbd-filter).
A refused open never leaves a volume read-only. mount without -w answers
a device it cannot open for writing with a read-only mount and exit 0, so SHC
mounts with -w, and the volume’s fstab entry carries x-systemd.rw-only for
boot. Before it reads, formats or mounts a DRBD device, the node waits for
drbdsetup events2 to say it may promote, and it tries a refused promote
again with backoff. Each wait logs X503425DRBWAT, a mount that went through
after one logs X200425VOLRWM, and after two minutes the install fails with
X503425VOLRWF, which names the refusal and the resource’s state; the peer’s
kernel log names what held the device open. A volume root found mounted
read-only is unmounted and mounted again read-write.
On a host whose LINSTOR storage pool is an LVM thin pool, LVM runs
thin_check before it activates the pool. A hard power-off with writes in
flight leaves leaked blocks, which thin_check reports as an error, and LVM
then keeps the pool inactive, with every copy on it, until someone repairs it.
Let leaked blocks through in /etc/lvm/lvmlocal.conf:
global { thin_check_options = [ "-q", "--clear-needs-check-flag", "--ignore-non-fatal-errors" ]}lvmconfig global/thin_check_options shows it. The initrd activates the
host’s volume groups at boot with its own copy of /etc/lvm, so rebuild it
with update-initramfs -u -k all after the change.
LVM logs nothing to the journal by default, and the LINSTOR satellite keeps
lvchange’s errors in its own error reports. For thin_check’s status to
reach the journal, append this line to /etc/lvm/lvm.conf. LVM ignores
logging keys in lvmlocal.conf.
log { syslog = 1 level = 3 }shc doctor run --category storage warns on a node whose thin pools hold
volumes without the option, and names any pool LVM refused, with
thin_check’s exit status when the journal records it: 64 is leaked blocks,
anything else wants lvconvert --repair. The node logs X503366THNPOL when
a pool is refused and once more when its status becomes known, and
shc events ls -T error lists it.
A node that comes back from a power cut mounts its volumes again by itself.
At boot the fstab mount of a DRBD device can run while DRBD is still attaching
its disk and fail with “Can’t open blockdev”. Once the node’s lease loop
renews its lease on the volume, shcd mounts the volume’s root and then its
bind through the same fstab entries, logs X200368VOLRMD, and the node agent
starts the deployment. A mount that keeps failing is logged as
X503368VOLRMT with the mount’s own error, tried again every 15 seconds, and
named by storage.volume-mounts. A lease-loss restart mounts every volume
of the deployment it can and fails naming each one it could not, rather than
stopping at the first.
Every writer of /etc/fstab on a node (block volumes, NFS, CephFS) holds one
lock and an flock on the table from its read to its write, and writes it in
place, whole. A line holding NUL bytes from an older torn write is taken out
by the next rewrite and logged as X200088FSTTRN. When a volume’s root or
bind line is missing and the volume is not mounted, the lease loop writes the
line again from the attach, but only once the attach has settled (the
attachment reads attached, no mount or unmount of it is running) and the line
has been missing for two minutes. It logs X200089FSTRGN with the line it
wrote.
Picking a provider for a specific stack
Section titled “Picking a provider for a specific stack”SHC’s storage module
(modules/storage/, ce module)
picks a provider at install time based on:
- App requirement:
requires.storage: [local, nfs]in the app’sapp.yamlwhitelists which providers are acceptable. - Per-stack override:
shc install <app> --storage <provider>at install time forces a specific provider (must still be in the whitelist). - Node affinity: the stack can only use providers that are linked to its assigned node.
- Capability requirements: if the app or install flow needs snapshots or quotas, providers without those capabilities are filtered out.
- Operator default: if multiple providers qualify, SHC picks
the cluster default (configurable in
default.yaml).
If no provider qualifies, install fails with
X400937QNOSUP (quotas not supported) / X400942SNOSUP (snapshots
not supported) / X404914STGNOF (storage provider not found).
Health & troubleshooting
Section titled “Health & troubleshooting”Quick health check
Section titled “Quick health check”shc storage lsshc doctor run --category storage # structured diagnosticsThe storage module registers a storage.local diagnosis: it walks
every registered provider and checks that the path exists, is a
directory, and is a mount point. A missing or non-directory path is
an error; an existing path that is not a mount point is a warning.
Source: modules/storage/doctor.go in the ce module.
storage.thin-pools runs on every node and reads its LVM: an error for each
thin pool LVM left inactive while it holds volumes, and a warning when
global/thin_check_options lacks --ignore-non-fatal-errors. Source:
modules/storage/thin_pools.go.
storage.volume-mounts runs on every node and fails for each block volume
attached to it that is not mounted where its deployment expects it, naming
the deployment, the mountpoint and why (X503368VOLRMT). Source:
modules/storage/volume_mounts.go.
storage.fstab runs on every node and fails for each line of /etc/fstab
that mount skips as a parse error (NUL bytes, fewer than three fields) and
for each block volume attached to the node with no fstab line for its root
or its bind, naming the volume and its deployment. It warns for a day on
each lost line the lease loop wrote again. Source:
modules/storage/fstab_doctor.go.
storage.volume-locality runs once for the cluster and warns on each block
volume in use on a node that holds no copy of it, a LINSTOR diskless
resource, naming the volume, its deployment and mount, the LINSTOR resource
and the node. One named for more than a few minutes has no copy coming:
linstor resource-definition list-properties <volume> shows the auto-diskful
options and linstor error-reports list why LINSTOR did not make the copy.
Source: modules/storage/volume_locality.go.
storage.connection-reach runs once for the cluster and warns on each volume
connection whose location holds no node, naming the connection, its tenant and
the location as its credential writes it. A volume it makes reaches only the
node it is attached on, so its deployment cannot move and each consumer
colocated with it is held to that node. Source:
modules/storage/connection_reach.go.
Common issues
Section titled “Common issues”“Provider is accessible but stacks fail with X400942SNOSUP on
backup.”
→ The backing filesystem doesn’t support snapshots. Install stacks
on a provider whose capabilities include snapshots, or accept the
rsync-fallback path (slower, non-atomic).
“NFS mount keeps dropping on reboot.”
→ Check that shc storage nfs join has registered the mount in
the SHC-managed systemd unit. Distro-level /etc/fstab entries
bypass SHC and should be removed if present.
“CephFS mount requires ceph kernel module.”
→ Load it (modprobe ceph) and add to /etc/modules-load.d/.
SHC’s cephfs-enable checks for this and refuses if missing.
“Adding a provider fails with ‘name is reserved’.”
→ local, nfs, and cephfs are reserved for managed singletons.
Use shc storage nfs enable / cephfs enable instead of
storage add nfs.
Disabling a provider
Section titled “Disabling a provider”Deleting a provider refuses if any stack still uses it. A relocation keeps
a deployment’s storage provider, so take each stack off it another way:
shc clone <stack> <new-name> --storage <other> and uninstall the original,
or reinstall it. Then:
shc storage rm <name>For the managed singletons, use the disable commands; they tear down the server/mount infrastructure as well:
shc storage nfs disable # CAREFUL: removes the cluster-wide mountshc storage cephfs disable # unmounts Ceph on all nodesDisable refuses if any stacks are still using the provider.
Related
Section titled “Related”- Multi-node cluster setup: plan storage before adding nodes
- Upgrading SHC: storage considerations during upgrade
- Architecture: storage module: internals
- App definition: how apps declare storage requirements
- CLI reference: storage: full command-line reference
(see also
shc storage nfs --help,shc storage cephfs --help)