Losing a site
Rig proof in progress. The steps below follow the code on
devwith SHC-1121 (the automatic voter count, shc!496 and ce!241) and SHC-1106 (control-plane is a node property:shc node update <node> --control-planeandshc node join --control-plane) merged. A two-site rig cell has run them six times (SHC-1124), the last from the loss to three voters again with every deployment running and every node answering on 443. Where a step says “on the rig”, that is what it measured. A claim still marked unproven is one the rig has not settled.
This page is for the day the site that holds the voters is gone: a datacenter fire, a provider account closed, every node in one location destroyed. For one lost node see Rotating nodes; for quorum loss inside one site see Disaster recovery.
The shape
Section titled “The shape”The voters live in the control-plane fence. It starts as the genesis
node’s location, and the voter count follows it: the largest odd number up
to its enrolled members, capped at five (one member gives one voter, two
give one voter and a non-voter, three or four give three, five or more
give five). A node joins as a voter only inside the fence and only while
the fence’s voters are below that target; every other joiner is a
non-voter, whatever order the nodes arrived in. shc label show control-plane lists the members; shc cluster status shows the voters
against them.
The leader runs a suffrage pass once a minute. Each pass makes at most one
change, a promotion before a demotion: a reachable non-voter inside the
fence is promoted while the fence’s voters are below target, a voter
outside the fence or above the target is demoted. A change in flight
holds every other membership change until it finishes. Its driver renews
it while it works; once it has gone five minutes unrenewed the next pass
takes it over and finishes it. Nothing expires on its own: a change on a
node that is out of raft and cannot take part (unreachable, or no longer
eligible) holds every other change until that node comes back or is
removed with shc cluster rm, which is what step 5 does for the dead
nodes.
Losing the whole home site therefore leaves the survivors with no voter. The control plane cannot commit: no writes, no placement, nothing reschedules. What still runs:
- containers already running on the survivors, until something needs the control plane;
- an app whose volume is far (its reach holds no control-plane fence node) is lease-exempt and keeps its writer through the loss;
- every other deployment with a single-writer volume on a survivor stops
itself. Its node renews the attachment lease every 5 s (TTL 15 s); after
five renewals fail in a row, about 25 s, the node stops the volume’s
writers (
docker stopwith a 10 s grace, then a kill), unmounts lazily and flushes. Expect those containers gone within about a minute of the loss, not within the TTL.
Before seating anything, confirm the stop happened on each survivor with
docker ps: no container of a stopped deployment may still run. That
local check is the one to trust here. shc ls, the deployment list,
reads through the control plane, so it is no witness until a voter is
seated again. A writer still up on a survivor is a writer the recovery
below must not race.
Losing one voter is not this page. shc cluster rm <node> removes it, and
the next suffrage pass promotes a non-voter inside the fence.
1. Confirm the site is gone
Section titled “1. Confirm the site is gone”Unreachable is not a synonym for dead. If the home site is only cut off, its voters are still a majority of the configuration they hold, and seating a second voter elsewhere forks the control plane. Before anything else:
- Check the home nodes out of band: the provider console, IPMI, the datacenter. Powered off or destroyed is gone. Unanswering is not.
- If the site can come back, wait for it. Raft resumes where it stopped and nothing below is needed.
- If it is gone, make sure it stays gone: power the home nodes off or detach them from every network. Two of three old voters coming back together are a majority of the old configuration, elect a leader and commit on their own.
2. Choose the survivor to seat
Section titled “2. Choose the survivor to seat”Pick one surviving node. Prefer the one with the newest raft log, the
highest applied_index in its rqlite status. rqlited’s HTTP API listens
on the node’s cluster address, port 4001, and on a cluster with TLS on it
demands a cluster-CA leaf from the caller, so run this on the node itself
with its own certificate:
sudo curl -s \ --cacert /var/lib/shc/certs/ca.pem \ --cert /var/lib/shc/certs/node.pem \ --key /var/lib/shc/certs/node-key.pem \ https://<this-node-cluster-address>:4001/status | jq '.store.raft.applied_index'Only when the cluster runs without TLS is it plain
curl -s http://<this-node-cluster-address>:4001/status.
On that node, keep a cold copy of the raft state before touching it. The
daemon runs as the shc systemd unit and rqlited runs inside it; both
must be stopped first, or the copy is a snapshot of a store mid-write.
The copy runs only when the stop succeeded and no rqlited is left:
sudo systemctl stop shc \ && ! pgrep -x rqlited > /dev/null \ && sudo cp -a /var/lib/shc/rqlite "/var/lib/shc/rqlite.before-site-loss-$(date +%s)" \ && echo "cold copy taken"sudo systemctl start shcNo cold copy taken line means no copy: find out why the stop failed or
which rqlited is still running, and go no further until a copy exists.
Wait until shc node status answers on that node before going on.
3. Seat it as the one voter
Section titled “3. Seat it as the one voter”On the chosen survivor:
shc cluster repair --force --yesshc cluster repair --force --dry-run previews this step: it asks every
question the recovery asks and prints the node it would seat and the
members it would drop, or the refusal, and changes nothing.
This writes a one-member raft configuration naming this node, as a voter,
and restarts its engine (how).
It refuses while any old voter answers (X409403CLSPRT) and when it cannot
tell (X409407CLSRVU); neither refusal is bypassable. It refuses too when
the node holds no raft state at all (X500201CLSREC): a wiped node cannot
be seated, only enrolled again.
Can it seat a node that was a non-voter? Yes. The recovery writes this node into the new configuration as a voter whatever its suffrage was, and nothing in the gate reads it. On the rig a far node that had only ever been a non-voter was seated this way twice: the command returned in 4 s and the node led a one-voter cluster about 20 s later, once the restart it schedules had run.
Check the result:
shc cluster statusVOTERS 1, this node the leader, writes working. The dead nodes still
show in the registry as unreachable, and the control-plane view warns that
the seated voter is outside the fence. Both are fixed below.
The vault on the seated node
Section titled “The vault on the seated node”Only a node that holds the auto-unlock credential file, usually the genesis
node, keeps the vault’s master password on disk. Every other node comes up
locked after a restart and takes the unlock from a peer that still holds
it, and those peers went with the site. The cold
copy in step 2 restarted this node, so until it is seated its daemon
answers from its boot store, which holds no vault state:
shc vault status reads INITIALIZED false and LOCKED true, and
shc vault unlock answers X503902VLTNAV. That is expected, and it is
why shc cluster repair --force is let past the vault lock.
Once the node is seated and shc has restarted onto raft, the vault’s
state is readable again. Unlock it with your own copy of the master
password, the one given to shc init or to the last shc vault rotate:
shc vault unlock --password-file /root/vault.pwshc vault statusLOCKED false, and a write commits. Nothing on the surviving nodes can
stand in for that password. To have this node unlock by itself at its
next start, now that no peer can do it, plant the sealed credential with
shc vault coldstart replant --node <this node> (see
the sealed auto-unlock credential).
4. Bring the other survivors back into raft
Section titled “4. Bring the other survivors back into raft”The new configuration names the seated node alone, so the other
survivors are no longer raft members, and their join markers point at the
dead site. Each one rejoins with its own token: a token is consumed by
the first enrolment that presents it, and a second node reusing it is
refused with X401022TKNCNS.
A rejoin keeps the node’s row, and with it its deployments and fence
memberships, only when it comes back under its existing name from
its existing cluster address. Enrolment adopts the old id only when
both match the row; a join without --name takes a fresh worker-…
name, and the same name from another address is treated as a different
machine. Both make a new row, while deployments and fences still name
the old one. So before the rejoin, note on each survivor its name, its
id and the interface it joined on:
shc node statussudo cat /var/lib/shc/node.idFor every other survivor, on the seated node:
shc node tokenthen on that survivor, as root, with the name it already has and, if it
was joined with --nic, the same interface:
sudo shc node join --force --name <its-name> [--nic <its-interface>] <token>Run as root, the join restarts the shc unit itself, and that restart
is what makes the daemon drop its old raft state and adopt the new
cluster. If the join instead prints Restart the daemon so it adopts the new cluster identity, restart it by hand:
sudo systemctl restart shcThe way back into raft is that restart. The join writes the seated
node’s raft address into cluster_join.json under the state home, with
the suffrage enrolment chose; on the restart the daemon starts rqlited
with -join against it (and -raft-non-voter for a non-voter), and
rqlited keeps retrying for about two minutes. Nothing else adds the node
back to raft, so a join whose restart did not happen leaves it out.
Before the next survivor, check that this one converged. On it,
sudo cat /var/lib/shc/node.id prints the id it had before. That alone
proves nothing about raft: the join writes the file before the restart,
and shc cluster status counts registry rows, which the survivor never
left. The proof is the live raft roster on the seated node, read with
its own certificate as in step 2, where the survivor’s id must appear
and be reachable:
sudo curl -s \ --cacert /var/lib/shc/certs/ca.pem \ --cert /var/lib/shc/certs/node.pem \ --key /var/lib/shc/certs/node-key.pem \ "https://<seated-node-cluster-address>:4001/nodes?ver=2&nonvoters=true" \ | jq '.nodes[] | select(.id == "<survivor-id>") | {id, voter, reachable}'No output, or "reachable": false, means the restart did not bring it
into raft: look at its daemon log before going on. Unproven: that
the row, its deployments and its fence memberships all survive the
rejoin. The rig checks it.
rqlite is the only raft. Once it commits again there is no second cluster to recover.
5. Move the control-plane fence
Section titled “5. Move the control-plane fence”Put the survivors in the fence before the dead nodes leave it. The fence refuses to be emptied, and the voter target counts its enrolled members:
shc node update <seated> --control-plane=trueshc node update <survivor> --control-plane=trueThen remove each dead node:
shc cluster rm <dead-node> --yesIt refuses (X400017NODDEP) while deployments still name the node.
Restore or move those apps to the surviving site first, or pass --force
to leave their rows naming a node that is gone and deal with them after.
Removal marks the node’s row removed, a tombstone the node list no
longer shows, and releases its voter slot; the row itself stays, which
matters if the node ever comes back (step 7).
From here the suffrage pass does the rest, one change a minute: it promotes non-voters inside the fence until the count reaches its target (two members give one voter, three give three). Watch it:
shc cluster statusNew nodes at the surviving site join the control plane when they join:
shc node join <token> --control-plane--control-plane puts the joiner in the control plane. --label key=value
(repeatable, one value per key) puts an operator label on it. Enrolment
decides suffrage after that: a voter only inside the control plane and
below its target, a non-voter otherwise.
6. Stateful apps
Section titled “6. Stateful apps”Apps whose volumes lived at the lost site come back from backups. See
Disaster recovery and the location-loss restore
runbook (SHC-1164). A far volume at the surviving site kept its writer
(it is lease-exempt); check the deployment is still running and leave it
alone. Every other single-writer deployment on the survivors stopped
itself during the loss (see The shape). Its node wrote the stop down in
single-writer-stops.jsonl under the state home, and starts the writer
again by itself once it holds a lease from a control plane that commits.
For a survivor other than the seated one that is its rejoin in step 4,
and nothing is typed for the deployment. On the rig the single-writer
fixture stopped 31 s after the home site lost power, and its container
started 12 s after its node’s shc node join --force returned, 39 s
after the join was typed.
Check each one with shc ls once its node is back in raft. While a
writer waits, its row ends with replica <service>/<n>: <entry> was stopped when this node's lease lapsed and starts again once the node holds its lease. One that still reads degraded two minutes after the
rejoin is not waiting for anything; start it with
shc restart -s <stack>.
7. When the old site comes back
Section titled “7. When the old site comes back”Never let a node from the lost site rejoin with its old raft state. Wipe
it and enrol it again, from a token minted on the current cluster, one
token per node, as root, with the --nic or --public-address it was
first joined with, if it had one:
sudo shc node join --force --name <name> [--nic <its-interface>] [--public-address <its-address>] <token>A join that reads the wrong address as the node’s own is refused
X403285ENRNAT, and the refusal names the --public-address to pass.
With neither flag the join reads the node’s own address from an
interface that is up, and passes over every bridge the node’s runtime
made.
Step 5 left its row as a tombstone, and the name decides what happens to
it. Enrolment finds a row by name, tombstones included, and adopts its
id when the joining machine’s address matches the row’s: the old name
from the old address revives the removed row, old id and all, and
every deployment shc cluster rm --force left naming that node points
at it again. Restore or move those deployments first, or give the node
a new name, which makes it a new member with a new id. After the join,
check which one you got: sudo cat /var/lib/shc/node.id on the node
against the id it had before the loss, and the deployments that name
it. As in step 4, the join restarts the shc unit; if it prints the
restart hint instead, run sudo systemctl restart shc. Before enrolling
the next, check on a current voter that the node’s id is in the live
raft roster and reachable, with the /nodes call from step 4. It joins as a non-voter unless it passes --control-plane or the control plane’s
home is its location again, and the suffrage pass decides from there.
A returned node gets its replica of every global service by itself, the
system Traefik among them, whether it came back under its old id or a new
name. On the rig shc ls read Traefik running on all five nodes two
minutes after the last join returned, and each returned node answered on
443 from then on. Until it does, that node serves no route.
Why the voters sit in one site
Section titled “Why the voters sit in one site”SHC-1050 §5.9 weighed the layouts. Three voters in the home site keep a quorum through a WAN cut and cost no cross-site write latency; losing the site costs this page. A 2 + 1 split puts a voter in the far site, which removes the question in step 3 but lets raft elect the far voter, after which every write crosses the WAN.
Adding a far node to the fence does not make it a voter: four members still give a target of three, which the home voters already fill, and the pass promotes only below target. Moving a voter is a deliberate operator act, never automatic. To place one at the far site:
shc node update <far-node> --control-plane=trueshc node update <home-voter> --control-plane=falseThe fence now holds three members, two of them voters, so it is below its target of three. A promotion comes before a demotion: the next pass promotes the far node, which takes the cluster to four voters for one pass, and the pass after demotes the home node, now a voter outside the fence, back to three. Then check the placement, and which node leads:
shc cluster statusLeadership is not moved for you: raft elects whichever voter wins, and a far leader means every write crosses the WAN until the home voters win it back.
Rig checklist (SHC-1123)
Section titled “Rig checklist (SHC-1123)”What the rig has to show before the unproven marks come off, with the log on SHC-1124:
- The home site’s nodes powered off; the far survivors hold no voter and the control plane refuses writes.
- A single-writer deployment on a far survivor stops itself within about a minute of the loss, and starts again by itself within two minutes of its node’s rejoin in item 5; a far-volume deployment keeps running.
- The stopped-daemon copy of
/var/lib/shc/rqliteis taken with rqlited exited, and the daemon comes back afterwards. With an rqlited kept running on purpose, the block refuses: no copy and nocold copy takenline. shc cluster repair --forceon a far non-voter seats it as the one voter, or fails with a named refusal.- One token per survivor:
sudo shc node join --force --name <its-name>brings each other survivor back into raft: the same id in/var/lib/shc/node.idbefore and after, that id reachable in the seated node’s/nodes?ver=2&nonvoters=trueroster, and its row, deployments and fence memberships intact. A rejoin without--namemakes a new row, which is the failure this step exists to avoid. - The fence move and the removals leave the suffrage pass promoting to the target, one change a minute.
- Adding a far node to the fence promotes nothing; removing a home voter from the fence promotes the far node on the next pass (four voters) and demotes the home node on the pass after (three).
- A home node that comes back wiped joins as a non-voter. Under a new
name it gets a new id; under its old name from its old address it
revives its removed row with the old id, and a deployment left naming
it by
shc cluster rm --forcenames it again. - A suffrage change interrupted on a home node when the site died holds
the pass (“waits for it to come back or be removed” in the pass’s
result) until
shc cluster rmremoves that node, and the pass then carries on.