Skip to content
SHC Docs

Storage module

Layer 5. Objects. Objects

The storage module owns the storage provider inventory, the managed-provider control plane (NFS and CephFS singletons), and the capability matrix (quotas, snapshots, filesystem detection) that higher-level features read.

NOT IMPLEMENTED: providers do not hold deployment state. A deployment’s compose file, DATA mounts, generated files, and app definition are written under the node’s own state home: paths.DeploymentsDir(), which is always <state_home>/deployments/.... No provider is consulted for that path.

storageapi.Provider declares BasePath() and CreateDeploymentFolder(), and local, nfs and cephfs each implement them, but nothing in either repo calls either method outside a unit test. Registering an NFS or CephFS provider mounts shared storage and records it in the inventory; it does not move any deployment onto it.

Whether to make provider-held deployment state real or to retire the interface is an open question; see the deployment binding model proposal, Q1 and slice 6.

KindCardinalitySharedManagedBase path
localmanynono/var/lib/shc (or configured)
nfs1 per clusteryesyes/mnt/shc/nfs
cephfs1 per clusteryesyes/mnt/shc/cephfs

local is the only kind you can instantiate multiple times; each entry points at an existing mount on specific nodes and is registered as external (managed = false). NFS and CephFS are singletons with reserved names; the enable/disable lifecycle is the only way to create or remove them.

Paths are relative to the pinned ce module.

modules/storage/
├── register.go # module registration + lifecycle wiring
├── service.go # StorageService (business logic)
├── router.go # method_router endpoints at /api/method/shc.storage.*
├── models.go # the store and store_node table constants
├── interfaces.go # the seams the service consumes
├── shim.go # registers local / NFS / CephFS providers, persists
│ # the store rows
├── shared_manager.go # SharedManager: cluster-singleton bookkeeping (NFS)
├── shared_manager_cephfs.go # the CephFS half of the same lifecycle
├── leader_forward.go # leader-aware dispatch for the shared-storage mutations
├── connbridge.go # consumes a connection's vault-backed volume credentials
├── crons.go # scheduled storage jobs
├── doctor.go # `shc doctor` checks for storage providers
├── volume_service.go # `shc volume ls/rm/rebind`
├── volume_repo.go # block-volume + attachment rows
├── volume_stonith.go # default (unconfigured) STONITH agent
├── volume_rebind_gate.go # the volume-rebind edition gate
├── storageapi/ # Provider interface, error codes, provider names
├── volumeapi/ # block-volume registry, STONITH seam, types
├── quotas/ # quota enforcement (xattr, XFS project, ZFS, scan)
├── snapshots/ # snapshot engines (btrfs, zfs, rsync)
├── hostrun/ # host command execution helper
├── providers/ # one package per backend
│ ├── local/ # local FS; also owns filesystem + capability detection
│ ├── nfs/ # NFS
│ ├── cephfs/ # CephFS
│ ├── volume/ # shared block-volume plumbing
│ ├── cloud/ # blank-import aggregator for the cloud backends
│ └── awsebs/ azuredisk/ cinder/ cloudstackvol/ dovolume/ gcepd/
│ hcloud/ linodevolume/ vultrvolume/
├── ceph/ # Ceph cluster primitives (internal plumbing)
│ ├── cmd.go # RunCeph / RunSystemctl subprocess helpers
│ ├── config.go # CephClusterConfig struct
│ ├── fsid.go # cluster fsid resolution
│ ├── health.go # CephHealthMonitor
│ ├── osd.go # CephOSDManager
│ ├── mon.go # CephMonitorManager
│ ├── mds.go # CephMDSManager
│ ├── cephadm.go # cephadm wrapper (containerised daemons, SSH orch)
│ ├── bootstrap.go # multi-node bootstrap (leader side)
│ ├── peer.go # peer enrollment (SSH key + labels over mesh)
│ ├── client.go # mon resolution / cephx client material
│ ├── leave.go # node-leave teardown
│ ├── platform_guard.go # host-platform preconditions
│ ├── util.go # shared helpers
│ └── manager.go # CephFSManager (orchestrator)
├── nfs/ # NFS exports + cleanup
│ ├── exports.go
│ └── cleanup.go
└── commands/ # CLI entrypoints
├── main.go # `shc storage`
├── add.go # `shc storage add`
├── list.go # `shc storage ls`
├── show.go # `shc storage show`
├── delete.go # `shc storage rm`
├── link.go # `shc storage link` + `shc storage unlink`
├── managed.go # `shc storage {nfs,cephfs} {enable|disable|status|join|leave}`
└── daemon_client.go # daemon transport for the above

The ceph/ subpackage is deliberately a wrapper around real Ceph primitives. Files in it are named after the Ceph daemon they manage and call ceph / systemctl directly via ceph/cmd.go. Callers outside the subpackage must go through CephFSManager; the service and router never reach for ceph subprocesses.

The two words are not interchangeable:

WordMeaning in SHCWhere it appears
cephLow-level cluster primitives (OSD, MON, MDS, MGR)modules/storage/ceph/, subprocess args, systemd units, xattrs, config defaults
cephfsThe SHC-managed shared filesystem productProvider kind, CLI root (shc storage cephfs), API routes (shc.storage.cephfs.*), request/response Go structs, mount path

The orchestrator class is CephFSManager (the product) even though it composes CephOSDManager, CephMonitorManager, CephMDSManager, and CephHealthMonitor (the primitives). Anything user-facing is cephfs; anything talking to the actual cluster is ceph.

All endpoints are POST, registered on method_router, and live at /api/method/:

EndpointPurpose
shc.storage.addRegister external local provider
shc.storage.listList providers (filterable by type/node)
shc.storage.getShow one provider
shc.storage.deleteRemove provider (force opt-in)
shc.storage.linkAttach external provider to a node
shc.storage.unlinkDetach external provider from a node
shc.storage.nfs.enableBring up managed NFS
shc.storage.nfs.disableTear down managed NFS
shc.storage.nfs.statusNFS server + mount state
shc.storage.nfs.joinPin NFS server role to a node
shc.storage.nfs.leaveRelease NFS server role
shc.storage.cephfs.enableBring up managed CephFS (multi-node bootstrap)
shc.storage.cephfs.disableTear down managed CephFS
shc.storage.cephfs.statusCluster/MDS/OSD health
shc.storage.cephfs.joinContribute an OSD
shc.storage.cephfs.leaveRemove an OSD (optionally forced)
shc.storage.pool.lsEvery storage pool and how full it is

All endpoints are guarded by RequirePermissions(...) with platform:manage or stack:read and scoped to the tenant context.

Every block provider (LINSTOR, Incus and each cloud’s) mounts a volume’s filesystem root at <state home>/block-volumes/<volume id> on the node holding it. Each node runs the node-local cron storage_usage once a minute: it statfs’s the roots the lease loop says it holds, its own state home, and, where its ceph CLI answers, the SHC-managed Ceph cluster and its OSDs. It upserts one kv row of its own (namespace storage.usage, keyed by node id, expiring after five minutes). Nothing measures on a request.

  • shc.storage.volume.ls and shc volume ls carry used_bytes, available_bytes and usage_measured_at per volume (the table prints USED and USE%; -o json carries the bytes). All three are absent for a volume no node measured in the last five minutes.
  • The stack read’s disk_bytes is the sum of its volumes’ used_bytes, absent when none was measured, and the metrics rollup publishes it as shc_stack_disk_usage_bytes.
  • shc.storage.pool.ls and shc storage pool ls (platform admin) list four kinds of pool: linstor (each storage pool on each node, from the controller of every volume connection, memoised for 30 s), ceph (the cluster and its OSDs), runtime (the pool each node’s runtime makes instance roots on, the row placement already keeps) and local (each node’s state home filesystem). A pool’s total_bytes, the table’s SIZE, is its size as df prints it, a filesystem’s root reserve included. A local pool’s used_bytes and free_bytes are df’s Used and Avail; a runtime pool’s used is its size less its free bytes, since placement’s row keeps no more, so on ext4 it counts the root reserve as used. USE% is used over used plus free, rounded up, as df’s Use% is.
  • shc doctor: storage.volume-usage and storage.pool-usage both warn at 80% and fail at 95%. A volume’s percent is used over used plus available, as df counts it. storage.pool-usage also names a pool source it could not read.

Both NFS and CephFS follow the same lifecycle shape:

stateDiagram-v2
[*] --> absent
absent --> enabled: enable
enabled --> absent: disable
enabled --> enabled: join/leave (capacity or role changes)
enabled --> degraded: node failure
degraded --> enabled: rebalance / re-elect
  • NFS: enable installs kernel NFS on a single voter node, exports /mnt/shc/nfs, and mounts it on every other node. join pins the server role to a specific node; leave releases that pin and re-elects a server.
  • CephFS: enable bootstraps a Ceph cluster (MON/MGR on voters, MDS on voters), creates a filesystem, and mounts it at /mnt/shc/cephfs. join --device adds a local OSD; leave --osd-id drains and removes one. disable tears that Ceph cluster down.

Automated multi-node CephFS enable is implemented via cephadm: the leader bootstraps the cluster (ceph/bootstrap.go, ceph/cephadm.go), distributes the cephadm SSH key + placement labels to peers over the mesh (ceph/peer.go, daemon/services_ceph_peer.go), and brings up MON/MGR/MDS placement before creating the filesystem and mounting /mnt/shc/cephfs. The whole path is unit-tested through the injected Runner, but it carries VALIDATION CAVEAT comments for a reason: the real bootstrap flags, SSH-over-mesh plumbing, and kernel mounts are only truly exercised on live nodes, and no rig scenario runs them yet; treat it as unvalidated until one is green on a ≥3-node cell.

LeaveLocalNode in modules/storage/ceph/leave.go tears down this node’s Ceph footprint when the node leaves: drain local OSDs, remove the local mon, stop the local mds, and stop the local ceph targets. Failures are warnings on LocalLeaveResult.

The local provider (providers/local/provider.go) maps on-disk filesystem types to SHC capability flags:

FilesystemQuota methodSnapshot method
XFSproject quotarsync fallback
Btrfssubvolume quotabtrfs snapshots
ZFSdataset quotazfs snapshots
CephFSxattr quotacephfs snapshots
NFSnone (scan only)rsync fallback
ext4/otherscan onlyrsync fallback

Provider rows carry quota_method and snapshot_method so StackService can decide at install time whether a deployment can satisfy its quota/snapshot requirements or must fall back to scan/rsync.

shc install --storage <name> records the name. It does not place anything.

The value is written to the deployment.storage column and to the storage field of the deployment’s manifest.json sidecar. ReadManifest does not read a state.json beside it. Two things read it back:

  1. The clone / move classifiers (daemon/clone_move_adapters.go, daemon/clone_dead_move_adapters.go) read it as the backend behind any persistent host-bind mount that carries no block-volume row; an empty value reads as local. The label is not proof of a provider. Both carry a ProviderProven fact, set only inside the branch that read a Volume row AND got a non-empty provider column off it, and a shared-filesystem verdict (Rebindable live, DeadMountShared on the dead-node path) requires it. Live, an unproven nfs / cephfs label degrades to NotRebindable, which forces the whole deployment onto backup+restore and so carries the bytes. On the dead-node path a label with no row at all is DeadMountLost, refused unless --discard-local names the loss; a label whose row names no provider instead falls through to the block/topology rules, which fail closed to DeadMountUnreachable. Each case names the unverifiable claim in the plan warnings, so the slower answer reads as deliberate. Note the consequence of the paragraph below: a shared label always mismatches the resolved block provider, so the binder never writes a row for one, and at HEAD those shared arms are therefore unreachable in production.

  2. BuildVolumeBinder (daemon/volume_binder.go) returns nil (the host-bind path) when a non-empty --storage names something other than the resolved block-volume provider. Only one of its four construction sites can see the flag. The daemon-served install threads it (modules/app/install_async.go → app.ResolveVolumeBinder → the VolumeBinderProvider seam the daemon fills at boot). The other three pass a hardcoded empty string: shc install (daemon/services_seams_stack.go, whose own comment says --storage “isn’t plumbed onto StackInstallRequest yet”), the restore-time provision attacher (daemon/services_backup_config.go), and shc volume rm|rebind (daemon/volume_ops.go). shc install, shc update and shc uninstall go through shcd. With shcd unreachable they refuse with X503738DMNREQ and change nothing. shc update never reaches the resolve at all, because install_async.go:139 is the seam’s ONLY call site in either repo, so it has no binder to decline with in the first place. No comment states a reason for the update path’s omission; treat it as unstated, not as a decision.

    The decline is also conditional on there being a block-volume backend to decline: with storage.volumes.provider unset and no volume-capable connection, the binder is nil for every install and --storage selects nothing. And the resolved provider is always a block backend, so an nfs or cephfs label mismatches it and the deploy stays on the host bind with no Volume row written, the fact the move classifiers were rebuilt around.

Nothing else consults it. In particular the deployment’s files land in <state_home>/deployments/... either way.

Omitting --storage leaves the column empty, which the classifiers read as local. There is no tenant, environment or cluster default. ResolveDefaultStorage and StorageConfig.default_storage are not in the tree. --storage records a name. It does not place the deployment. Whether that name becomes a placement input is the deployment binding model proposal, Q1.