Losing a location
A location is lost when every node of one location label is gone and is not coming back: the site burned, the provider will not return the hardware, or you have stopped waiting. Its deployments come back from their off-site archives into a surviving location. You lose whatever changed since each deployment’s last archive, so the recovery point is the backup schedule.
This page assumes the control plane survived. If the lost location held a
majority of the voters, shc cluster status says so, and
Disaster recovery comes first: restore quorum, then
come back here.
What has to be true before it happens
Section titled “What has to be true before it happens”- Every location label names an off-site repo, one that lives outside
that location. A deployment’s archives go to its location’s repo unless a
narrower scope sets
backup.repo.shc label ls -vshows theBACKUP-REPOcolumn; set a missing one withshc label set location=<provider>/<location> --backup-repo <repo>. - The mirror copies each of those repos to a second provider every night,
at
<backup/copy_repo>/locations/<provider>/<location>(a location with no provider sits under_). See 6b in Enable backups. bitspur’s second provider is Backblaze B2. - Every deployment is on a schedule. A deployment with no archive has no recovery.
1. Confirm the location is gone, and list what ran there
Section titled “1. Confirm the location is gone, and list what ran there”shc ls lists only the environment $SHC_ENVIRONMENT or the context
(shc ctx -e) selects, even with --all-tenants, so clear both before the
inventory or the lost deployments of every other environment never reach
the list. shc ctx shows the context; when it names an environment, reset
it, and select the cluster again if it named one:
unset SHC_ENVIRONMENTshc ctxshc ctx --resetshc ctx --cluster bitspurshc node lsshc cluster statusshc label show location=gorillaservers/ogdenshc ls --all-tenants > ogden-deployments.txtEvery node of the location reads lost in the CONDITION column.
shc cluster status shows the voters against the control-plane fence; if
fewer than a majority are healthy, stop and go to
Disaster recovery.
shc label show names the location’s member nodes. The deployments to
restore are the rows of shc ls --all-tenants whose node, or any
entry of whose nodes, is one of those members. The label value’s own deployment
list is not enough: it holds only the deployments a constraint, a store or a
volume ties to the label, not everything that happened to run there.
2. Choose the node that takes them
Section titled “2. Choose the node that takes them”Run every step from here on a survivor node. The recover is placed the way an install is and runs on the node it lands on, which need not be the one you run it from: the archives go there and the data replays there. Check the surviving location has room:
shc node statusshc label show location=hetzner/hel1Room includes disk for the deployments’ images. A recover lands only on a node that can take the images it lacks, and refuses before it installs anything when no node can, naming the one that comes closest (step 5).
An app whose provider (its database, say) survived in another location is outside this page: leave it off, stop for it and escalate. Its recover runs the app’s install hooks against the live provider before the app’s own files are back, with no restore protection on that provider, and an app’s init can drop tables it does not recognise as its own.
A deployment on a block volume is outside this page as well. A recover keeps the volume the deployment already has, wherever it lands, and a cloud volume in the lost location attaches to no node anywhere else (Hetzner refuses a server in another location). List the block volumes:
shc volume ls --all-tenants -o json > ogden-volumes.txtA row whose tenant, environment and stack match a deployment on the
list means leave that deployment off, stop for it and escalate. Data on host
binds and Docker named volumes is not a block volume, is not listed here, and
comes back from the archive.
If the lost location held the estate’s only ingress entry point, relocate
that first by changing where it may run. Its durable volumes died with the
node, so it comes back from its latest backup; it re-points every managed DNS
record at the new node and needs a DNS provider to do it. --where replaces
the deployment’s whole expression, so a term naming the lost node does not
stay behind and leave no node. The update prints its plan before it asks:
shc update -t master traefik -e default --where 'node=hel-02' --restore-from-backupWith an edge per location, the survivor’s edge already serves, and this step does not apply.
3. Take each deployment’s archive out of its repo
Section titled “3. Take each deployment’s archive out of its repo”Restore only the deployments on the list from step 1, less those step 2 set aside. A deployment that runs in a surviving location is never restored: replaying an older archive over it would roll back its live data.
A deployment is a tenant, an environment and a stack, and the archive is
chosen by all three. backup ls filters by stack only, and its table shows
neither the environment nor the consistency id:
mkdir -p /var/lib/shc/recoveryshc backup ls -t <tenant> -s nextcloud -o jsonshc backup ls -t <tenant> -s nextcloud-db -o jsonUse the completed rows of that stack. An app and its database that are both on the list recover together, in one call, and only from archives that carry the same consistency id, or neither carries one: a scheduled group fire gives every member the same id, and the recover refuses inputs with two different ones. So take the newest app archive whose id also appears on a database archive, and that database archive with it, even when one of them has a newer archive alone: the pair is the recovery point.
A pair also has to share its tenant and environment. One recover call
installs and restores every archive into the tenant named by -t and the -e it is given, so a
prod app whose database lives in another environment, or another tenant,
would get that database restored under the wrong deployment. The export
below lets you check it from each archive’s own manifest.
If no pair shares an id, or the app’s database sits in another tenant or environment, stop and escalate; this page has no safe path for it. Recovering them one at a time is not one: the database’s restore protection ends when its own recover finishes, and the app’s recover then runs its install hooks before its files are back, and an app’s init can drop tables it does not recognise as its own.
Verify each chosen archive and export it to a file on this node. A backup
id is looked up in the tenant named by -t.
backup verify reads an archive of any size: it streams, so a large archive
costs time and egress, not memory. Its refusal is about the archive itself
(no data, a missing subject, a container that does not add up), and only
that is a reason to take an older one:
shc backup verify -t <tenant> 20260930T031500Z_645cafff7f65e1d6shc backup export -t <tenant> -i 20260930T031500Z_645cafff7f65e1d6 -f /var/lib/shc/recovery/nextcloud.bakshc backup verify -t <tenant> 20260930T031500Z_9d1e2f3a4b5c6d7eshc backup export -t <tenant> -i 20260930T031500Z_9d1e2f3a4b5c6d7e -f /var/lib/shc/recovery/nextcloud-db.bakThe export reads the archive from the lost location’s off-site repo with that deployment’s settings.
Before recovering a pair, read both manifests. Each .bak is an ar archive
(ar comes with binutils) whose manifest.json names the deployment it
came from and the providers it was wired to:
ar p /var/lib/shc/recovery/nextcloud.bak manifest.json | jq '{tenant, environment, consistency, integrations: [.integrations[]? | {provider_stack_name, provider_tenant_name, provider_environment}]}'ar p /var/lib/shc/recovery/nextcloud-db.bak manifest.json | jq '{tenant, environment, consistency}'Both archives name the same tenant, environment and consistency (or
neither has one), and the app’s integration with the database carries no
provider_tenant_name or provider_environment: those appear only when the
provider sat in another tenant or environment. Any difference, or a manifest
you cannot read, means stop and escalate.
4. Replace the rules that name the lost location
Section titled “4. Replace the rules that name the lost location”A recover is placed by the deployment’s rules, so a deployment whose placement rule names the lost location, or one of its nodes, cannot come back until that rule is replaced. List the rules that name the location and those that name each of its nodes:
shc constraint ls --label location=gorillaservers/ogden --allshc constraint ls --label node=ogden-01 --allReplace each deployment’s rule with the survivor’s location, naming the deployment by its tenant, stack and environment:
shc constraint set -t <tenant> -s nextcloud -e prod --where 'location=hetzner/hel1' --scope deployment --reason 'ogden lost 2026-09-30'A rule set at tenant or cluster scope binds every deployment under it, and
a narrower rule cannot loosen it: replace it at its own scope, with a where
that keeps the survivor. shc constraint explain -t <tenant> -s nextcloud -e prod
shows which scope set each rule.
5. Recover
Section titled “5. Recover”Dry-run first, then recover, with the deployment’s environment. The pair
from step 3 goes in one call, and the database comes up before the app. The
dry run names the node the pair lands on (node_name, and node_reason for
why):
shc recover -t <tenant> -b /var/lib/shc/recovery/nextcloud.bak -b /var/lib/shc/recovery/nextcloud-db.bak -e prod --dry-runshc recover -t <tenant> -b /var/lib/shc/recovery/nextcloud.bak -b /var/lib/shc/recovery/nextcloud-db.bak -e prod --yesA deployment with no provider at all recovers alone, the same way.
The recover is placed the way an install is: the deployments’ rules, each
node’s condition, its cpu and memory, and its disk for the images the archived
deployments run and it does not hold. Every deployment of one call lands on
one node, so the app and its database come back together.
It is never bound to the lost node. When the node it lands on is not the one you run it from,
the archives go to that node streamed to its disk, and that node runs the
install and the data replay; the operation’s progress names it. The
deployment’s row still names the dead node and moves to the node it lands on.
The deployment keeps its name. A deployment whose rules leave out the node it
would land on is refused with X409173FNCRCV, naming each rule and the
command that replaces it, and nothing is installed: go back to step 4 for it,
and the dry run says the same.
A call no node has the disk for, keeping 2 GiB free, is refused with
X507299RCVNRM, which names the node that comes closest, and nothing is
pulled. Free disk there, or recover the deployments in separate calls when
they have no provider between them.
A node or an image the check cannot measure, an incus node or an image from a
registry that wants credentials, is recovered without it, and the recover says
so on a warning line (X200297RCVRUM).
The recover replays each deployment’s ingress, and managed DNS records follow it to the new node. If that replay fails, the log says so and names the update that finishes it.
Three commands look like they fit and do not:
shc recover --consistency-idrestores each member in place, which needs the dead node, and a scheduled fire’s group can include deployments that are still running elsewhere.shc backup restorerestores into a stack that already runs on this node.shc update --whererelocates a deployment off a dead node by reusing the volumes that survive it. A volume in the lost location cannot attach anywhere else, so the update is refused naming it, and--restore-from-backuprestores only the deployment’s latest backup in its own repo.
6. Give the recovered deployments a backup home
Section titled “6. Give the recovered deployments a backup home”They now back up to the survivor location’s repo, unless a narrower scope
names one: a backup.repo set at the tenant, environment, stack or
deployment wins over the location. Check those first, and unset or change any that names the lost
location’s repo:
shc config get -t <tenant> backup.repo --scope deploymentshc config get -t <tenant> backup.repo --scope stackshc config get -t <tenant> backup.repo --scope environmentshc config get -t <tenant> backup.repo --scope tenantshc config get backup.repo --scope deployment -t <tenant>shc config unset -t <tenant> backup.repo --scope deploymentThe recover restored the deployment’s own settings from the archive, so a
deployment-scope backup.repo is the likeliest to still name the lost repo.
Check the survivor’s location label names a repo, set it if not (outside that location), take the first archive in the new place and read its row:
shc label show location=hetzner/hel1shc label set location=hetzner/hel1 --backup-repo s3:https://fsn1.your-objectstorage.com/backups-hel1shc backup create -t <tenant> -s nextcloud -e prodshc backup verify -t <tenant> 20260930T120000Z_1f0c9e2a7b3d4e5fshc backup ls -t <tenant> -s nextcloud -o jsonThe location names the survivor’s repo. Until that archive exists, the only copy of this deployment’s data is the archive you recovered from.
7. Forget the lost nodes
Section titled “7. Forget the lost nodes”Once every deployment on the list is recovered:
shc node forget ogden-01 --yesshc node forget ogden-01 --force --yesThe first form refuses (X400017NODDEP) while rows still name the node.
On a lost node the refusal names the second form and no drain: --force
removes the node and leaves the rows naming it as they are. Keep the lost
location’s fence. It carries the old repo, so the archives taken there stay
listed and restorable, and the mirror keeps copying them.
When the location’s off-site repo is gone too
Section titled “When the location’s off-site repo is gone too”Then the archives come from the second-provider mirror. The mirror is a file
copy, so the repo password is the same; the credentials are the second
provider’s, which the daemon only reads from backup.* settings. Set those
first, at global scope, and at any narrower scope that overrides them for a
deployment you are restoring:
shc config get backup.aws_access_key_id --scope stackshc config set backup.aws_access_key_id <copy-key-id> --scope globalshc config set backup.aws_secret_access_key <copy-secret> --scope globalThe keys are the vault’s backup/copy_aws_*. The relocate opens the mirror
with the global key and these credentials.
Then point every row that names the lost repo at its mirror copy. The mirror
runs nightly, so it lacks whatever was captured after its last run, and
those archives went with the repo. --skip-missing moves only the rows whose
snapshot the mirror holds and leaves the rest naming the lost repo; the dry
run says how many it leaves:
shc backup relocate --from s3:https://ord.example.com/backups-ogden --to s3:https://s3.us-west-004.backblazeb2.com/shc-backup-copy/locations/gorillaservers/ogden --skip-missing --dry-runshc backup relocate --from s3:https://ord.example.com/backups-ogden --to s3:https://s3.us-west-004.backblazeb2.com/shc-backup-copy/locations/gorillaservers/ogden --skip-missing --yesIn step 3, take the newest archive the relocate moved, the newest row whose
location names the mirror: a newer row it left naming the lost repo cannot
be read.
Run steps 3 to 5 while the keys are set. They switch every repo that inherits them, so put the originals back before the next scheduled fire.
Then carry on from step 6.
Proving it
Section titled “Proving it”The two-site rig runs this page as scenario S10: it cuts a location, follows
the steps, and the log goes on SHC-1164.
tests/unit/cliverbs/location_loss_runbook_test.go checks that every command
here resolves, every flag exists, and the recovery keeps to backup export
and recover -b ….
See also
Section titled “See also”- Disaster recovery: quorum loss, dead leaders
- Enable backups: repos, schedules, the off-provider mirror
- Rotating nodes in and out