Skip to content
SHC Docs

Restoring the whole cluster

This page assumes the app ladder has run on this cluster, so it holds web, db, pgweb, placed, store and cloud. It backs them all up into archives outside the cluster, wipes every machine’s SHC state, builds a fresh cluster on the same four machines and recovers every app from the archives. The platform’s own deployments are not backed up: a fresh init makes them again.

The app ladder’s pages 01 to 07 left placed held to learn/home. Move it to learn/far first, as that ladder’s node-failure page does, so the restore below brings back a deployment whose rule keeps it to one node.

home01
$ shc update placed --where location=learn/far --yes
...
$ shc ls
...

cloud reads db and store through its plugs, and pgweb reads db. A backup of one of them alone is a database without its app, or an app without its database. --all-connected backs up the whole integration graph at one point in time, each member into the repository, and writes one group archive of all of them under /var/lib/shc/exports. The archive is in the clear, so that a cluster with a different key can read it.

home01
$ shc backup create cloud --all-connected
...
$ shc backup ls -s cloud --limit 1
...

Export the group as one archive: -s cloud takes cloud’s latest backup, the row above, and --with-connected adds every member of its group. /srv/handover is outside SHC’s state, so the wipe below leaves it alone; on a real cluster copy the directory off the machine.

home01
$ shc backup export -s cloud --with-connected --file /srv/handover/cluster.bkg
...

placed and web stand alone. Seal each backup’s secrets under a password of your own, so that a cluster whose vault differs can open them, and write the latest one out as a file named for its deployment.

home01
$ shc backup create placed --secrets-password P@ssw0rd
...
$ shc backup export -s placed --latest --file /srv/handover/placed.bak
/srv/handover/placed.bak
$ shc backup create web --secrets-password P@ssw0rd
...
$ shc backup export -s web --latest --file /srv/handover/web.bak
/srv/handover/web.bak
$ ls -1 /srv/handover
a1.bak
cluster.bkg
placed.bak
web.bak

a1.bak is the repositories page’s leftover; it stays, unused.

placed is limited to learn/far, and its archive carries that rule. A recover lands a deployment on the node it is typed on, so placed’s archive has to be on far01 too. Write it there as well. No tenant is selected on far01, so the export names prod with -t.

far01
$ mkdir -p /srv/handover
$ shc backup export -s placed --latest -t prod --file /srv/handover/placed.bak
/srv/handover/placed.bak

The app ladder’s app directories under ~/learn on home01 and the notes app under /usr/share/shc/apps are files of yours, not SHC state. They stay too.

shc node leave --purge takes a node out of the cluster and destroys every bit of SHC state on it: the database, the certificates, every deployment’s containers, networks and volumes. Do the three joined nodes first, then home01, which is by then a cluster of one.

far01
$ shc node leave --purge --force --yes
...
home03
$ shc node leave --purge --force --yes
...
home02
$ shc node leave --purge --force --yes
...
home01
$ shc node leave --purge --force --yes
...

A leave that fails part-way is picked up where it stopped with --resume; none fails here, so it is not typed.

explained, not walked: no leave fails here
$ shc node leave --resume --yes

Nothing of the cluster is left on any machine. The archives are.

home01
$ ls -1 /srv/handover
a1.bak
cluster.bkg
placed.bak
web.bak

The same four nodes, the same names, locations and labels, the same vault password, this time read from files only root can read. --force would wipe a node that still held a cluster; this one holds none. --modules-config names the platform modules, here the four the daemon installs by default. On these two-CPU machines init skips openobserve, as it did on the first init. placed’s rule names learn/far and comes back with it, so far01 must be there again.

~/vault-password
P@ssw0rd
~/backup-password
P@ssw0rd
home01
$ chmod 600 ~/vault-password ~/backup-password
$ shc init --name home01 --cloud learn --location home --label tier=core --force --modules-config '{"postgres":{},"keycloak":{},"openobserve":{},"traefik":{}}' --vault-password-file ~/vault-password --backup-password-file ~/backup-password --yes
...
$ shc license install ~/shc-license.token
licence installed
state: keyed
tier: pro
...
home02
$ ssh home01 shc node token --ttl 30m --fields token -o text | shc node join - --name home02 --cloud learn --location home --control-plane
✔ joined cluster as node "home02" at 192.168.232.12
home03
$ ssh home01 shc node token --ttl 30m --fields token -o text | shc node join - --name home03 --cloud learn --location home --control-plane
✔ joined cluster as node "home03" at 192.168.232.13
far01
$ ssh home01 shc node token --ttl 30m --fields token -o text | shc node join - --name far01 --cloud learn --location far --label tier=edge
✔ joined cluster as node "far01" at 192.168.232.14
home01
$ shc node ls
NAME STATE CONDITION AVAIL CLOUD LOCATION ROOM BUILD STORE LEADER VOTER
far01 healthy ready active learn learn/far ok 0.1.47 replica no no
home01 healthy ready active learn learn/home ok 0.1.47 voter yes yes
home02 healthy ready active learn learn/home ok 0.1.47 voter no yes
home03 healthy ready active learn learn/home ok 0.1.47 voter no yes
$ shc ls -t master
STACK TENANT ENV APP STATUS NODES
keycloak master default keycloak running home01
postgres master default postgres running home01
traefik master default traefik running far01, home01, home02, home03

The fresh cluster holds master alone, and home01’s context went with the old cluster’s state. Add prod again and select it, so the recovered deployments land in it. The catalogue apps come from the public repository, which shc init registered again as shc. The backup repository is set again too, at global scope as on the repositories page, so that the recovered deployments can be backed up from here on.

home01
$ shc tenant add prod --no-admin
✔ created tenant "prod"
warning: keycloak realm provisioning still in progress
$ shc ctx -t prod
...
$ shc repo ls
...
$ shc config set backup.repo /var/backups/shc/repo --scope global
backup.repo is a path: unless every node mounts the same storage at /var/backups/shc/repo, each of this cluster's 4 nodes keeps its own repository there. A backup is then on the node that took it alone and is lost with that node
KEY VALUE SCOPE
backup.repo /var/backups/shc/repo global

shc recover installs deployments from archives, with their variables, their integrations and their data. The archives were sealed by the old cluster’s vault, so give it that vault’s password, on the command line or from the file. --dry-run prints the plan and does nothing: --rename would bring a deployment back under another name, --override under another ingress host, --skip-integration without one of its plugs, and --runtime and -e name the runtime and environment it lands in, here the ones it had. Read the plans, then recover as it was.

home01
$ shc recover -b /srv/handover/web.bak --rename web=web2 --runtime docker --dry-run
...
$ shc recover -f /srv/handover/cluster.bkg --secrets-password P@ssw0rd --override cloud:hosts=cloud.learn.example --skip-integration s3 --dry-run
...
$ shc recover -f /srv/handover/cluster.bkg --secrets-password-file ~/vault-password --dry-run
...
$ shc recover -f /srv/handover/cluster.bkg --secrets-password-file ~/vault-password --yes
...
$ shc recover -b /srv/handover/web.bak --secrets-password P@ssw0rd --yes
...

placed’s archive carries its rule, where location=learn/far. A recover is placed the way an install is, and home01, where it is typed, is not in learn/far, so placed lands on far01, the node the rule keeps: the archive goes there and its data replays there. The dry run names the node.

home01
$ shc recover -b /srv/handover/placed.bak --secrets-password P@ssw0rd --dry-run
...
$ shc recover -b /srv/handover/placed.bak --secrets-password P@ssw0rd --yes
...

Three forms of shc recover are not typed on this ladder. --consistency-id recovers a group straight from the repository, with no archive, which a wiped cluster no longer has. --storage puts every recovered deployment on one storage backend instead of the one each backup recorded. -e names the environment the deployments land in.

explained, not walked: the wipe leaves no repository, one backend, the default environment
$ shc recover --consistency-id 20261008T170012Z_5c1d3e4f5a6b7c8d --storage local -e prod --yes

A control plane restored from an older backup meets nodes that hold a newer peer table and refuse it; shc cluster confirm-restore makes the restored one authoritative. This cluster was rebuilt, not restored, so it is not typed.

explained, not walked: this cluster was rebuilt from archives, no control plane was restored from an older backup
$ shc cluster confirm-restore --yes

Run shc ls until every deployment is running.

home01
$ shc ls
STACK TENANT ENV APP STATUS NODE
cloud prod prod nextcloud running home01
db prod prod postgres running home01
pgweb prod prod /root/learn/pgweb running home01
placed prod prod notes running far01
store prod prod /root/learn/versitygw running home01
web prod prod /root/learn/nginx running home01
...
$ shc integration ls
...

placed is on far01 again, and the rule that keeps it there came back with it.

home01
$ shc constraint explain -s placed
...

The data came with them. nginx serves its page, placed has its row, and Nextcloud has the file the app ladder wrote.

home01
$ curl -s http://localhost:18080/
<!doctype html>
<title>hello</title>
<p>Hello from SHC.</p>
$ shc exec -s placed -S postgres -- psql -U postgres -tAc 'select body from notes'
kept by the backup
$ curl -s -u admin:P@ssw0rd http://localhost:8082/remote.php/dav/files/admin/hello.txt
Hello from SHC.

The cluster is as the app ladder left it, on machines that held nothing an hour ago.

Next: The daemon’s settings.