Skip to content
SHC Docs

shc storage

The shc storage commands manage the storage provider inventory: which shared filesystems exist, which nodes can reach them, and the lifecycle of the two SHC-managed kinds. SHC supports three provider kinds:

KindShared across nodesManaged by SHCTypical path
localNoNo/var/lib/shc or custom
nfsYesYes (singleton)/mnt/shc/nfs
cephfsYesYes (singleton)/mnt/shc/cephfs

Registering a provider does not move any deployment onto it. Deployment files always live under the node’s own state home (<state_home>/deployments/...); no provider is consulted for that path. shc install --storage <name> records a label on the deployment, and only the clone/move classifiers and the block-volume binder read it back; see Using a provider in a deployment below and the storage module architecture.

  • Managed providers are brought up and torn down by SHC. There is at most one of each kind (nfs, cephfs) per cluster, and their names are reserved.
  • External providers are local-kind entries that point at an existing mount. They are the only kind you create with shc storage add.
  • Links associate a provider with specific nodes. Managed providers link themselves; external providers are linked via shc storage link.

List all storage providers known to the cluster.

Terminal window
shc storage ls
shc storage ls --type cephfs
shc storage ls --node node01
shc storage ls --managed
shc storage ls --external

Options:

FlagShortDescription
--typeFilter by provider kind (local, nfs, cephfs)
--node-nFilter to providers linked to a node
--managedShow only SHC-managed providers
--externalShow only external providers

Default columns are name, type, filesystem, managed, quota, snapshot, nodes (count of linked nodes), and path. Output format follows the global -o/--output flag (tty|text|json|yaml).

Show full details for one provider.

Terminal window
shc storage show local-ssd

Returns the provider id, path, filesystem, quota and snapshot capability, config blob, and the list of linked nodes.

Register an existing mount as an external local-kind provider.

Terminal window
shc storage add local-ssd --path /var/lib/shc-ssd
shc storage add local-ssd --path /var/lib/shc-ssd --node node01

Options:

FlagShortDescription
--path-pMount path on the target node (required)
--node-nNode to link; defaults to no link (link later with shc storage link)

Rules:

  • The name must not be one of the reserved managed-provider names local, nfs, cephfs.
  • For managed providers, use shc storage nfs enable or shc storage cephfs enable instead.

Remove a provider from the inventory.

Terminal window
shc storage rm local-ssd
shc storage rm local-ssd --force

By default the delete is refused if any stack still uses the provider. --force skips that check.

Attach or detach an external provider to one or more nodes.

Terminal window
shc storage link local-ssd node01 node02
shc storage unlink local-ssd node01

Both commands take the provider name followed by one or more node ids. Managed providers manage their own links.

Managed NFS is a single kernel NFS server exported from one cluster node, mounted by every other node at /mnt/shc/nfs. It is created and torn down by SHC.

Bring up managed NFS on the cluster.

Terminal window
shc storage nfs enable
shc storage nfs enable --export-path /mnt/shc/nfs --network 10.42.0.0/24

Options:

FlagDefaultDescription
--export-path/mnt/shc/nfsBase export path on the server node
--network(auto)CIDR allowed to mount the export

Tear down managed NFS. Refused (X409033STGUSE) while any deployment’s storage column or any volume row still names the provider (the same in-use join shc storage rm runs). The refusal names the stacks holding it.

Terminal window
shc storage nfs disable
shc storage nfs disable --force # tear down anyway; those stacks lose the share

--force orphans the referencing rows: nothing reaps them, and the deployments keep pointing at a provider that no longer exists. Move or uninstall them first unless you are recovering a cluster whose consumers are already gone.

Show NFS server role, export path, and per-node mount state.

Terminal window
shc storage nfs status
shc storage nfs status -v

Pin the NFS server role onto a node.

Terminal window
shc storage nfs join # pin local node
shc storage nfs join --node node02 # pin a specific node

Release the NFS server role from the pinned node. SHC will re-elect another voter as server.

Terminal window
shc storage nfs leave
shc storage nfs leave --node node02

Managed CephFS gives the cluster a replicated, distributed filesystem mounted at /mnt/shc/cephfs. Each node that contributes capacity runs one or more Ceph OSDs backed by a block device. Monitors and the MDS are placed on voter nodes.

Enable CephFS across the cluster. Requires at least one voter node.

Terminal window
shc storage cephfs enable

Note: multi-node enable (key distribution, monmap bootstrap, MDS/MGR deployment) is not yet automated. The command is wired up end-to-end and will succeed once the deployment path lands.

Tear down CephFS, stopping local monitors, MDS, and OSDs and removing the mount. Refused (X409033STGUSE) while any deployment’s storage column or any volume row still names the provider (the same in-use join shc storage rm runs).

Terminal window
shc storage cephfs disable
shc storage cephfs disable --force # tear down anyway; those stacks lose the mount

Show cluster health, OSD counts, and per-node daemon state.

Terminal window
shc storage cephfs status

Contribute a local block device as a new OSD.

Terminal window
shc storage cephfs join --device /dev/sdb
shc storage cephfs join --device /dev/sdb --node node03

Options:

FlagShortDescription
--deviceBlock device to consume (required, e.g. /dev/sdb)
--node-nNode to add the OSD on; defaults to local

Remove an OSD from the cluster, waiting for rebalance by default.

Terminal window
shc storage cephfs leave --osd-id 3
shc storage cephfs leave --osd-id 3 --force

Options:

FlagShortDescription
--osd-idOSD id to remove (required)
--force-fSkip the wait for rebalance

shc node leave automatically drains and removes every local OSD before the node exits the cluster, so manual cephfs leave is only needed for capacity management.

--storage is repeatable on shc install, and each value takes one of three forms:

ValueMeaning
<backend> or nodeevery volume of the deployment goes there; node is the node’s own disk
<volume>=<size>one volume’s size, on the deployment’s backend
<volume>=<backend>:<size>one volume’s backend and size
Terminal window
shc install nextcloud --storage cephfs
shc install nextcloud --storage local-ssd
shc install nextcloud --storage nfs --stack cloud --environment prod
shc install postgres --storage pgdata=50Gi
shc install postgres --storage hcloud --storage pgdata=50Gi
shc install postgres --storage pgdata=hcloud:50Gi --storage wal=node:10Gi
shc install postgres --storage node --storage pgdata=hcloud:50Gi

A backend is node, the configured block-volume provider, a volume-capable connection, or a provider shc storage ls lists. Any other name is refused with X400305STGBKE, which lists the backends the cluster has. A size takes Ki, Mi, Gi, Ti, K, M, G or T, the units an app’s x-shc-volumes size takes, and a size for a volume the app does not declare is refused with X400304STGVOL. Each volume binds on the backend its value names, so one deployment can hold a volume on hcloud, another on ceph and others on the node disk. With --storage node, every volume not named on another backend stays on the node disk. The deployment records each volume’s backend, and a later render keeps the volume there. A backend the cluster has but that cannot bind a block volume, such as a provider the cluster’s volume configuration pins away from, is refused with X400307STGBND. A volume on node stays on the node disk even when the app declares x-shc-volumes for it. A volume the deployment already keeps on a block volume is refused with X409305VOLNOD instead: a render does not move its bytes.

A volume with no size on the command line takes the app’s x-shc-volumes size. A block volume is provisioned at its size. A volume on the node disk takes a size only where the node holding it can hold the volume to it: an xfs filesystem mounted with prjquota, a zfs dataset of its own or a btrfs subvolume with quotas on (the quota module says where). Before any volume is bound or prepared, the render asks that node, which may not be the node rendering, and refuses a size it cannot cap with X409311VOLQTA, naming the volume, the node and the filesystem. A size it takes is recorded on the deployment and becomes the volume’s hard cap at the next disk sweep.

shc: [X409311VOLQTA] volume gitlab-logs cannot be capped at 10Gi on node incus03 (ext4): ext4 has no directory quota

Before any image is pulled, admission charges every new volume’s size to the disk quota of the deployment, its environment and its tenant, and refuses an install that does not fit:

shc: [X403008DSKASK] disk quota of deployment pg exceeded, 40Gi in use, 50Gi asked, 64Gi limit

shc update takes --storage <volume>=<size> only, and a volume keeps its backend.

Terminal window
shc update postgres --storage pgdata=80Gi

A volume never shrinks. A larger size grows a block volume: its provider resizes it, the node it is attached to grows the filesystem on it online (ext4, xfs and btrfs) once the disk reads the new size, and then the volume’s record takes the size the provider reports. Hetzner volumes grow in whole GB, rounded up. Every size is checked before any volume grows, so one volume that cannot grow refuses the whole update with nothing changed. Its own size is no change and a smaller one is refused with X409303VOLSHR. A provider that cannot resize a volume refuses with X409306VOLRSZ, and a volume attached to no node with X409308VOLDET. A volume on the node disk takes the new size where the node holding it can cap it, and its cap moves to it at the next disk sweep. A size below the bytes it holds is refused with X409303VOLSHR before anything changes, and a size its node cannot cap with X409311VOLQTA before the render binds or prepares any volume. Sizes print in the units the flag takes, so a size in a refusal can be typed back as it reads.

shc: [X409303VOLSHR] volume pgdata is 80Gi, an update cannot shrink it to 50Gi
shc: [X409306VOLRSZ] volume pgdata is on ceph, which cannot grow a volume

An app update that raises a block volume’s x-shc-volumes size grows the volume the same way. Where the volume cannot grow, the update goes ahead, the volume stays at its size, and the update warns X200303VOLSIZ: the app now sizes volume pgdata at 80Gi, it stays at 50Gi.

If a block volume has no recorded size, the update reads it from the provider and records it before deciding whether to grow. If the size cannot be recorded, the update continues and warns X200304VOLUNK with the volume’s requested size and the reason.

On each leadership gain, the leader sweeps live block volumes with no recorded size and asks their providers one at a time, leaving kept or released volumes alone. Doctor’s storage.volume-size-recorded check lists any live volumes still without a size; a provider that cannot report one needs shc update --storage <volume>=<size>.

The deployment’s backend is stored on the deployment: the deployment.storage column, and the storage field of the deployment’s manifest.json sidecar. node is stored as local, the store’s name for the node disk. Two consumers read it back:

  • Relocation (shc update --where, live or off a dead node) reads it, and since the Slice 0 fix it no longer believes it on its own. An nfs or cephfs label buys a rebindable (live) or shared (dead-node) verdict only when a persisted Volume row also names that provider; nothing else at HEAD can say where a deployment’s bytes actually are. An unbacked label degrades to not-rebindable on the live path, which forces backup+restore and so carries the bytes; on the dead-node path a label with no row is treated as lost and refused unless --restore-from-backup brings the data back from the latest backup. Both say in the plan warnings that the claim could not be verified, so the slower answer reads as deliberate rather than as a fault.
  • The block-volume binder, on the daemon-served shc install only, declines when --storage names something other than the resolved block-volume provider, which leaves the deploy on the host-bind path. This has an effect only on an estate that has a block-volume backend at all; with none configured the binder is absent either way and --storage changes nothing. That decline is not a substitute for checking the persisted volume records. Install, update and uninstall require a reachable shcd.

The deployment’s files land in <state_home>/deployments/... regardless: no value of --storage moves them, and no provider is consulted for that path.

Omitting --storage leaves the column empty. The relocation classifiers interpret an empty value as local.

CommandDescription
shc storage lsList providers
shc storage show <name>Show one provider
shc storage add <name> --path ...Register an external local provider
shc storage rm <name>Remove a provider
shc storage link <name> <node>...Link external provider to nodes
shc storage unlink <name> <node>...Unlink external provider from nodes
shc storage nfs enableBring up managed NFS
shc storage nfs disableTear down managed NFS
shc storage nfs statusShow NFS status
shc storage nfs joinPin NFS server role to a node
shc storage nfs leaveRelease NFS server role from a node
shc storage cephfs enableBring up managed CephFS
shc storage cephfs disableTear down managed CephFS
shc storage cephfs statusShow CephFS status
shc storage cephfs join --deviceContribute a disk as an OSD
shc storage cephfs leave --osd-idRemove an OSD