Skip to content
SHC Docs

Storage providers

A storage provider is where a deployment’s volumes live. Every cluster starts with one managed local provider per node, named node:<node id>, the daemon’s state directory on that node, and SHC read the filesystem’s abilities when it set each one up.

This page links a provider to a second node and moves the NFS share’s server from one node to another, so it needs a second node. Prepare a second Debian 13 machine the way home01 was prepared, with 2 CPUs, 8 GB of memory and SHC installed, named home02. It must reach home01, and root on it must be able to ssh home01 with a key and no prompt: the join reads its token from home01 that way. The join asks home01 for a fresh token over ssh and hands it to shc node join -, which reads it from standard input.

home02
$ ssh home01 shc node token --fields token -o text | shc node join - --name home02 --cloud learn --location home
✔ joined cluster as node "home02" at 192.168.233.12

Run this again if home02 is still vault-locked or not ready. Within a minute of this first join home01’s daemon restarts once, as SHC promotes its database from one file into a replicated one; type a command again if it fails in that moment.

home01
$ shc node ls
NAME STATE CONDITION AVAIL CLOUD LOCATION ROOM BUILD STORE LEADER VOTER
home01 healthy ready active learn learn/home ok 0.1.47 voter yes yes
home02 healthy ready active learn learn/home ok 0.1.47 replica no no

Each node now has its local provider:

home01
$ shc storage ls
...
$ shc storage ls --type local
...
$ shc storage ls --managed
...
$ shc storage ls --external
...

A second local provider is a directory on a node, for a faster disk or a bigger one. The directory is made on the node that will serve it; the provider records the path and does not check it.

home01
$ mkdir -p /mnt/fast
home01
$ shc storage add fast --path /mnt/fast
...
$ shc storage show fast
...
$ shc storage ls
...
$ shc storage rm fast --yes
...

--node ties a provider to one node at add; shc storage link adds nodes to a provider, unlink takes them away, and shc storage ls --node lists the providers a node serves. Each takes a node name, as every other node flag does.

home01
$ shc storage add fast --path /mnt/fast --node home02
...
$ shc storage show fast
...
$ shc storage link fast home01
✔ linked storage provider "fast" to node home01
$ shc storage ls --node home01
...
$ shc storage unlink fast home01
✔ unlinked storage provider "fast" from node home01
$ shc storage rm fast --yes
...

shc storage rm refuses a provider a deployment is using unless --force says so. A provider can also be a cloud’s block storage, reached through a connection.

explained, not walked: nothing uses fast, and the learn cluster has no cloud block-storage connection
$ shc storage rm fast --yes --force
$ shc storage add cloud --connection <connection>

The managed NFS share is one export, served by one node and mounted on every other. A deployment that names it as its provider is counted as using it; the volumes themselves stay on node disk, as the install below says. The export lives under /mnt/shc/nfs on the server, the node the command is typed on. --bind makes the NFS daemon listen on one address of that node only, named as an IP address or as an interface: mesh0 is the node mesh interface, the same word on every node, and the server resolves it to its own address. An interface the server does not have is refused before anything is installed. --network says which addresses may mount it: 10.192.0.0/11 is the mesh range every node holds an address in, so the nodes and nothing else.

home01
$ shc storage nfs enable --network 10.192.0.0/11 --bind nosuch0
shc: [X503073PEERHTT] internal-API peer call failed 10.64.48.1:4003 storage.nfs_enable_server remote [X400081NFSBND] nfs server cannot listen on nosuch0, it is neither an IP address nor an interface with an IPv4 address on this node
$ shc storage nfs enable --network 10.192.0.0/11 --bind mesh0
...
$ shc storage nfs status
...
$ shc storage show nfs
...
$ shc storage ls --managed
...
$ shc storage logs nfs
...

shc storage logs nfs prints the last lines of the NFS services’ journal on the daemon’s host; --follow keeps following them. The export path has the default this cluster wants, so the page above did not name it.

explained, not walked: --follow runs until interrupted, and the default export path is what the page typed
$ shc storage logs nfs --follow
$ shc storage nfs enable --export-path /mnt/shc/nfs --network 10.192.0.0/11 --bind mesh0

Install the catalogue Postgres with the share named as its provider. --storage records the provider on the deployment: shc storage rm and shc storage nfs disable then count the deployment as using it and refuse without --force. The volumes themselves stay on the node disk; a provider named this way does not move them.

home01
$ shc install postgres -s pgnfs --storage nfs
installing deployment "pgnfs"
[pgnfs] Healthy 1/1
✔ postgres-1 healthy
✔ installed deployment "pgnfs"
$ shc ls
STACK TENANT ENV APP STATUS NODE
cloud prod prod nextcloud running home01
db prod prod postgres running home01
pgnfs prod prod postgres running home02
pgweb prod prod /root/learn/pgweb running home01
placed prod prod notes running home01
store prod prod /root/learn/versitygw running home01
web prod prod /root/learn/nginx running home01
db prod staging postgres running home01

The server role can move. shc storage nfs join pins a node as the server, bound to its own mesh interface like the first, and leave releases it.

home01
$ shc storage nfs join --node home02 --bind mesh0
✔ pinned nfs server to node "home02"
$ shc storage nfs status
ENABLED STATUS EXPORT PATH SERVER NODE
true ok /mnt/shc/nfs home02
$ shc storage nfs leave --node home02
✔ released nfs server role, server now "home01"
$ shc storage nfs status
ENABLED STATUS EXPORT PATH SERVER NODE
true ok /mnt/shc/nfs home01

Remove the deployment, then the share. shc storage nfs disable refuses while a deployment uses the share unless --force says so, and it unmounts every node.

home01
$ shc uninstall pgnfs --yes
uninstalling deployment "pgnfs"
[pgnfs] Complete 5/5
✔ remove from home02
✔ take the safety backup
✔ leave every node's service VIP table
✔ tear down
✔ remove the copy on home01
✔ uninstalled deployment "pgnfs"
$ shc storage nfs disable --yes
✔ disabled nfs storage
$ shc storage ls
...
home01
$ shc ls
STACK TENANT ENV APP STATUS NODE
cloud prod prod nextcloud running home01
db prod prod postgres running home01
pgweb prod prod /root/learn/pgweb running home01
placed prod prod notes running home01
store prod prod /root/learn/versitygw running home01
web prod prod /root/learn/nginx running home01
db prod staging postgres running home01

--force on nfs disable takes the share away from a deployment still using it. CephFS is the other managed share, with native snapshots: it needs a spare disk per node, each served by an OSD, the daemon that owns one disk. --via binds an OSD’s replication traffic to one address or interface; --force on leave skips the wait for the rebalance, and on disable ignores the deployments still using it.

explained, not walked: nothing used the share at disable, and the learn machines have no spare disk for CephFS
$ shc storage nfs disable --yes --force
$ shc storage cephfs enable --public-network 10.192.0.0/11 --cluster-network 10.192.0.0/11
$ shc storage cephfs join --node home02 --device /dev/sdb --via mesh0
$ shc storage cephfs status
$ shc storage cephfs leave --osd-id 0 --yes
$ shc storage cephfs leave --osd-id 0 --yes --force
$ shc storage cephfs disable --yes
$ shc storage cephfs disable --yes --force

shc storage typed alone answers for the whole group, and so do its two share groups.

home01
$ shc storage
...
$ shc storage nfs
...
$ shc storage cephfs
...

The cluster has its node providers again as the only ones, and tenant prod holds the deployments the environments page left.