Skip to content
SHC Docs

Losing a site

Rig proof in progress. The steps below follow the code on dev with SHC-1121 (the automatic voter count, shc!496 and ce!241) and SHC-1106 (control-plane is a node property: shc node update <node> --control-plane and shc node join --control-plane) merged. A two-site rig cell has run them six times (SHC-1124), the last from the loss to three voters again with every deployment running and every node answering on 443. Where a step says “on the rig”, that is what it measured. A claim still marked unproven is one the rig has not settled.

This page is for the day the site that holds the voters is gone: a datacenter fire, a provider account closed, every node in one location destroyed. For one lost node see Rotating nodes; for quorum loss inside one site see Disaster recovery.

The voters live in the control-plane fence. It starts as the genesis node’s location, and the voter count follows it: the largest odd number up to its enrolled members, capped at five (one member gives one voter, two give one voter and a non-voter, three or four give three, five or more give five). A node joins as a voter only inside the fence and only while the fence’s voters are below that target; every other joiner is a non-voter, whatever order the nodes arrived in. shc label show control-plane lists the members; shc cluster status shows the voters against them.

The leader runs a suffrage pass once a minute. Each pass makes at most one change, a promotion before a demotion: a reachable non-voter inside the fence is promoted while the fence’s voters are below target, a voter outside the fence or above the target is demoted. A change in flight holds every other membership change until it finishes. Its driver renews it while it works; once it has gone five minutes unrenewed the next pass takes it over and finishes it. Nothing expires on its own: a change on a node that is out of raft and cannot take part (unreachable, or no longer eligible) holds every other change until that node comes back or is removed with shc cluster rm, which is what step 5 does for the dead nodes.

Losing the whole home site therefore leaves the survivors with no voter. The control plane cannot commit: no writes, no placement, nothing reschedules. What still runs:

  • containers already running on the survivors, until something needs the control plane;
  • an app whose volume is far (its reach holds no control-plane fence node) is lease-exempt and keeps its writer through the loss;
  • every other deployment with a single-writer volume on a survivor stops itself. Its node renews the attachment lease every 5 s (TTL 15 s); after five renewals fail in a row, about 25 s, the node stops the volume’s writers (docker stop with a 10 s grace, then a kill), unmounts lazily and flushes. Expect those containers gone within about a minute of the loss, not within the TTL.

Before seating anything, confirm the stop happened on each survivor with docker ps: no container of a stopped deployment may still run. That local check is the one to trust here. shc ls, the deployment list, reads through the control plane, so it is no witness until a voter is seated again. A writer still up on a survivor is a writer the recovery below must not race.

Losing one voter is not this page. shc cluster rm <node> removes it, and the next suffrage pass promotes a non-voter inside the fence.

Unreachable is not a synonym for dead. If the home site is only cut off, its voters are still a majority of the configuration they hold, and seating a second voter elsewhere forks the control plane. Before anything else:

  1. Check the home nodes out of band: the provider console, IPMI, the datacenter. Powered off or destroyed is gone. Unanswering is not.
  2. If the site can come back, wait for it. Raft resumes where it stopped and nothing below is needed.
  3. If it is gone, make sure it stays gone: power the home nodes off or detach them from every network. Two of three old voters coming back together are a majority of the old configuration, elect a leader and commit on their own.

Pick one surviving node. Prefer the one with the newest raft log, the highest applied_index in its rqlite status. rqlited’s HTTP API listens on the node’s cluster address, port 4001, and on a cluster with TLS on it demands a cluster-CA leaf from the caller, so run this on the node itself with its own certificate:

Terminal window
sudo curl -s \
--cacert /var/lib/shc/certs/ca.pem \
--cert /var/lib/shc/certs/node.pem \
--key /var/lib/shc/certs/node-key.pem \
https://<this-node-cluster-address>:4001/status | jq '.store.raft.applied_index'

Only when the cluster runs without TLS is it plain curl -s http://<this-node-cluster-address>:4001/status.

On that node, keep a cold copy of the raft state before touching it. The daemon runs as the shc systemd unit and rqlited runs inside it; both must be stopped first, or the copy is a snapshot of a store mid-write. The copy runs only when the stop succeeded and no rqlited is left:

Terminal window
sudo systemctl stop shc \
&& ! pgrep -x rqlited > /dev/null \
&& sudo cp -a /var/lib/shc/rqlite "/var/lib/shc/rqlite.before-site-loss-$(date +%s)" \
&& echo "cold copy taken"
sudo systemctl start shc

No cold copy taken line means no copy: find out why the stop failed or which rqlited is still running, and go no further until a copy exists. Wait until shc node status answers on that node before going on.

On the chosen survivor:

Terminal window
shc cluster repair --force --yes

shc cluster repair --force --dry-run previews this step: it asks every question the recovery asks and prints the node it would seat and the members it would drop, or the refusal, and changes nothing.

This writes a one-member raft configuration naming this node, as a voter, and restarts its engine (how). It refuses while any old voter answers (X409403CLSPRT) and when it cannot tell (X409407CLSRVU); neither refusal is bypassable. It refuses too when the node holds no raft state at all (X500201CLSREC): a wiped node cannot be seated, only enrolled again.

Can it seat a node that was a non-voter? Yes. The recovery writes this node into the new configuration as a voter whatever its suffrage was, and nothing in the gate reads it. On the rig a far node that had only ever been a non-voter was seated this way twice: the command returned in 4 s and the node led a one-voter cluster about 20 s later, once the restart it schedules had run.

Check the result:

Terminal window
shc cluster status

VOTERS 1, this node the leader, writes working. The dead nodes still show in the registry as unreachable, and the control-plane view warns that the seated voter is outside the fence. Both are fixed below.

Only a node that holds the auto-unlock credential file, usually the genesis node, keeps the vault’s master password on disk. Every other node comes up locked after a restart and takes the unlock from a peer that still holds it, and those peers went with the site. The cold copy in step 2 restarted this node, so until it is seated its daemon answers from its boot store, which holds no vault state: shc vault status reads INITIALIZED false and LOCKED true, and shc vault unlock answers X503902VLTNAV. That is expected, and it is why shc cluster repair --force is let past the vault lock.

Once the node is seated and shc has restarted onto raft, the vault’s state is readable again. Unlock it with your own copy of the master password, the one given to shc init or to the last shc vault rotate:

Terminal window
shc vault unlock --password-file /root/vault.pw
shc vault status

LOCKED false, and a write commits. Nothing on the surviving nodes can stand in for that password. To have this node unlock by itself at its next start, now that no peer can do it, plant the sealed credential with shc vault coldstart replant --node <this node> (see the sealed auto-unlock credential).

4. Bring the other survivors back into raft

Section titled “4. Bring the other survivors back into raft”

The new configuration names the seated node alone, so the other survivors are no longer raft members, and their join markers point at the dead site. Each one rejoins with its own token: a token is consumed by the first enrolment that presents it, and a second node reusing it is refused with X401022TKNCNS.

A rejoin keeps the node’s row, and with it its deployments and fence memberships, only when it comes back under its existing name from its existing cluster address. Enrolment adopts the old id only when both match the row; a join without --name takes a fresh worker-… name, and the same name from another address is treated as a different machine. Both make a new row, while deployments and fences still name the old one. So before the rejoin, note on each survivor its name, its id and the interface it joined on:

Terminal window
shc node status
sudo cat /var/lib/shc/node.id

For every other survivor, on the seated node:

Terminal window
shc node token

then on that survivor, as root, with the name it already has and, if it was joined with --nic, the same interface:

Terminal window
sudo shc node join --force --name <its-name> [--nic <its-interface>] <token>

Run as root, the join restarts the shc unit itself, and that restart is what makes the daemon drop its old raft state and adopt the new cluster. If the join instead prints Restart the daemon so it adopts the new cluster identity, restart it by hand:

Terminal window
sudo systemctl restart shc

The way back into raft is that restart. The join writes the seated node’s raft address into cluster_join.json under the state home, with the suffrage enrolment chose; on the restart the daemon starts rqlited with -join against it (and -raft-non-voter for a non-voter), and rqlited keeps retrying for about two minutes. Nothing else adds the node back to raft, so a join whose restart did not happen leaves it out.

Before the next survivor, check that this one converged. On it, sudo cat /var/lib/shc/node.id prints the id it had before. That alone proves nothing about raft: the join writes the file before the restart, and shc cluster status counts registry rows, which the survivor never left. The proof is the live raft roster on the seated node, read with its own certificate as in step 2, where the survivor’s id must appear and be reachable:

Terminal window
sudo curl -s \
--cacert /var/lib/shc/certs/ca.pem \
--cert /var/lib/shc/certs/node.pem \
--key /var/lib/shc/certs/node-key.pem \
"https://<seated-node-cluster-address>:4001/nodes?ver=2&nonvoters=true" \
| jq '.nodes[] | select(.id == "<survivor-id>") | {id, voter, reachable}'

No output, or "reachable": false, means the restart did not bring it into raft: look at its daemon log before going on. Unproven: that the row, its deployments and its fence memberships all survive the rejoin. The rig checks it.

rqlite is the only raft. Once it commits again there is no second cluster to recover.

Put the survivors in the fence before the dead nodes leave it. The fence refuses to be emptied, and the voter target counts its enrolled members:

Terminal window
shc node update <seated> --control-plane=true
shc node update <survivor> --control-plane=true

Then remove each dead node:

Terminal window
shc cluster rm <dead-node> --yes

It refuses (X400017NODDEP) while deployments still name the node. Restore or move those apps to the surviving site first, or pass --force to leave their rows naming a node that is gone and deal with them after. Removal marks the node’s row removed, a tombstone the node list no longer shows, and releases its voter slot; the row itself stays, which matters if the node ever comes back (step 7).

From here the suffrage pass does the rest, one change a minute: it promotes non-voters inside the fence until the count reaches its target (two members give one voter, three give three). Watch it:

Terminal window
shc cluster status

New nodes at the surviving site join the control plane when they join:

Terminal window
shc node join <token> --control-plane

--control-plane puts the joiner in the control plane. --label key=value (repeatable, one value per key) puts an operator label on it. Enrolment decides suffrage after that: a voter only inside the control plane and below its target, a non-voter otherwise.

Apps whose volumes lived at the lost site come back from backups. See Disaster recovery and the location-loss restore runbook (SHC-1164). A far volume at the surviving site kept its writer (it is lease-exempt); check the deployment is still running and leave it alone. Every other single-writer deployment on the survivors stopped itself during the loss (see The shape). Its node wrote the stop down in single-writer-stops.jsonl under the state home, and starts the writer again by itself once it holds a lease from a control plane that commits. For a survivor other than the seated one that is its rejoin in step 4, and nothing is typed for the deployment. On the rig the single-writer fixture stopped 31 s after the home site lost power, and its container started 12 s after its node’s shc node join --force returned, 39 s after the join was typed.

Check each one with shc ls once its node is back in raft. While a writer waits, its row ends with replica <service>/<n>: <entry> was stopped when this node's lease lapsed and starts again once the node holds its lease. One that still reads degraded two minutes after the rejoin is not waiting for anything; start it with shc restart -s <stack>.

Never let a node from the lost site rejoin with its old raft state. Wipe it and enrol it again, from a token minted on the current cluster, one token per node, as root, with the --nic or --public-address it was first joined with, if it had one:

Terminal window
sudo shc node join --force --name <name> [--nic <its-interface>] [--public-address <its-address>] <token>

A join that reads the wrong address as the node’s own is refused X403285ENRNAT, and the refusal names the --public-address to pass. With neither flag the join reads the node’s own address from an interface that is up, and passes over every bridge the node’s runtime made.

Step 5 left its row as a tombstone, and the name decides what happens to it. Enrolment finds a row by name, tombstones included, and adopts its id when the joining machine’s address matches the row’s: the old name from the old address revives the removed row, old id and all, and every deployment shc cluster rm --force left naming that node points at it again. Restore or move those deployments first, or give the node a new name, which makes it a new member with a new id. After the join, check which one you got: sudo cat /var/lib/shc/node.id on the node against the id it had before the loss, and the deployments that name it. As in step 4, the join restarts the shc unit; if it prints the restart hint instead, run sudo systemctl restart shc. Before enrolling the next, check on a current voter that the node’s id is in the live raft roster and reachable, with the /nodes call from step 4. It joins as a non-voter unless it passes --control-plane or the control plane’s home is its location again, and the suffrage pass decides from there.

A returned node gets its replica of every global service by itself, the system Traefik among them, whether it came back under its old id or a new name. On the rig shc ls read Traefik running on all five nodes two minutes after the last join returned, and each returned node answered on 443 from then on. Until it does, that node serves no route.

SHC-1050 §5.9 weighed the layouts. Three voters in the home site keep a quorum through a WAN cut and cost no cross-site write latency; losing the site costs this page. A 2 + 1 split puts a voter in the far site, which removes the question in step 3 but lets raft elect the far voter, after which every write crosses the WAN.

Adding a far node to the fence does not make it a voter: four members still give a target of three, which the home voters already fill, and the pass promotes only below target. Moving a voter is a deliberate operator act, never automatic. To place one at the far site:

Terminal window
shc node update <far-node> --control-plane=true
shc node update <home-voter> --control-plane=false

The fence now holds three members, two of them voters, so it is below its target of three. A promotion comes before a demotion: the next pass promotes the far node, which takes the cluster to four voters for one pass, and the pass after demotes the home node, now a voter outside the fence, back to three. Then check the placement, and which node leads:

Terminal window
shc cluster status

Leadership is not moved for you: raft elects whichever voter wins, and a far leader means every write crosses the WAN until the home voters win it back.

What the rig has to show before the unproven marks come off, with the log on SHC-1124:

  1. The home site’s nodes powered off; the far survivors hold no voter and the control plane refuses writes.
  2. A single-writer deployment on a far survivor stops itself within about a minute of the loss, and starts again by itself within two minutes of its node’s rejoin in item 5; a far-volume deployment keeps running.
  3. The stopped-daemon copy of /var/lib/shc/rqlite is taken with rqlited exited, and the daemon comes back afterwards. With an rqlited kept running on purpose, the block refuses: no copy and no cold copy taken line.
  4. shc cluster repair --force on a far non-voter seats it as the one voter, or fails with a named refusal.
  5. One token per survivor: sudo shc node join --force --name <its-name> brings each other survivor back into raft: the same id in /var/lib/shc/node.id before and after, that id reachable in the seated node’s /nodes?ver=2&nonvoters=true roster, and its row, deployments and fence memberships intact. A rejoin without --name makes a new row, which is the failure this step exists to avoid.
  6. The fence move and the removals leave the suffrage pass promoting to the target, one change a minute.
  7. Adding a far node to the fence promotes nothing; removing a home voter from the fence promotes the far node on the next pass (four voters) and demotes the home node on the pass after (three).
  8. A home node that comes back wiped joins as a non-voter. Under a new name it gets a new id; under its old name from its old address it revives its removed row with the old id, and a deployment left naming it by shc cluster rm --force names it again.
  9. A suffrage change interrupted on a home node when the site died holds the pass (“waits for it to come back or be removed” in the pass’s result) until shc cluster rm removes that node, and the pass then carries on.