Skip to content
SHC Docs

Nodes failing and rejoining

A node is lost when its daemon stops answering. The leader marks it, and decides for each deployment on it: one whose data can go with it moves to a node its rules allow; one whose data is a local volume is held where it is and starts again when the node returns. a1, a2, a3 and a6 keep no data, so they can move.

Two reads before anything breaks: every node answers a ping, and the daemon’s own log on a node is read from any other.

home01
$ shc cluster ping -c 1 -T 5
...
$ shc cluster logs home02 -S shc -n 5
...

home02 votes. Stop SHC there. This stops the daemon, not the containers.

home02
$ systemctl stop shc

Run shc node ls on home01 until home02 changes state.

home01
$ shc node ls
NAME STATE CONDITION AVAIL CLOUD LOCATION ROOM BUILD STORE LEADER VOTER
far01 healthy ready active learn learn/far ok 0.1.47 replica no no
home01 healthy ready active learn learn/home ok 0.1.47 voter yes yes
home02 unreachable down active learn learn/home ok 0.1.47 voter no yes
home03 healthy ready active learn learn/home ok 0.1.47 voter no yes

Two of the three voters answer, so the cluster keeps writing. The leader’s own watch names the node it lost.

home01
$ shc cluster status
cluster name: shc-cluster
leader: home01
nodes: 3 healthy of 4, 1 unreachable
voters: 3
quorum: healthy, 2 voters required
warning: zero fault tolerance: 2 of 3 raft voters reachable, so losing one more stops the cluster committing, and removing a voter needs a quorum of the voters that remain, so restore the unreachable voters before touching membership or the cluster can no longer evict its own dead nodes
leader-only work, recorded by home01 22s ago:
...
$ shc leader health
NODE ID STATE OBSERVED AT ISSUE
...
node-8897fa9ce7cf unreachable 2026-10-09T05:15:15Z -
...

Read what the leader decided for the hello deployments. A row it has moved names its new node; one it holds names home02 and says why.

home01
$ shc ls --where owner=learn
...

Start SHC on home02 again. It comes back as a voter, and nothing moves back.

home02
$ systemctl start shc
home01
$ shc node ls
NAME STATE CONDITION AVAIL CLOUD LOCATION ROOM BUILD STORE LEADER VOTER
far01 healthy ready active learn learn/far ok 0.1.47 replica no no
home01 healthy ready active learn learn/home ok 0.1.47 voter yes yes
home02 healthy ready active learn learn/home ok 0.1.47 voter no yes
home03 healthy ready active learn learn/home ok 0.1.47 voter no yes

home01 leads and runs a1, whose rule allows home01 alone. Stop SHC there and read the cluster from home02.

home01
$ systemctl stop shc

The two voters left elect one of themselves. Run shc cluster status on home02 until it names a leader.

home02
$ shc leader status
...
$ shc cluster status
cluster name: shc-cluster
leader: home02
nodes: 3 healthy of 4, 1 unreachable
voters: 3
quorum: healthy, 2 voters required
warning: zero fault tolerance: 2 of 3 raft voters reachable, so losing one more stops the cluster committing, and removing a voter needs a quorum of the voters that remain, so restore the unreachable voters before touching membership or the cluster can no longer evict its own dead nodes
leader-only work, recorded by home02 54s ago:
...
$ shc node ls --where node=home01
NAME STATE CONDITION AVAIL CLOUD LOCATION ROOM BUILD STORE LEADER VOTER
home01 unreachable down active learn learn/home ok 0.1.47 voter no yes

a1 may run nowhere else, so the new leader holds it. The row keeps naming home01 and says why it is not running.

home02
$ shc ls --where stack=a1
STACK TENANT ENV APP STATUS NODE REASON
...
warning: 1 of 1 deployments are not running: 1 unknown
warning: node home01 is unreachable since 2026-10-08T20:17:03.313542101Z and holds 1 deployment

The a1 row reads unknown on home01, and its reason says when the node went unreachable and why failover found no node for it: each node’s condition or fence, in whatever order the leader lists them.

Start SHC on home01. It rejoins as a follower; the lead stays where the election put it, and a1 runs again where it was.

home01
$ systemctl start shc
home02
$ shc node ls --where node=home01
NAME STATE CONDITION AVAIL CLOUD LOCATION ROOM BUILD STORE LEADER VOTER
home01 healthy ready active learn learn/home ok 0.1.47 voter no yes
$ shc cluster status
cluster name: shc-cluster
leader: home02
nodes: 4 healthy of 4
voters: 3
quorum: healthy, 2 voters required
leader-only work, recorded by home02 47s ago:
...
home01
$ shc ls --where stack=a1
STACK TENANT ENV APP STATUS NODE
a1 prod prod /root/learn/cluster/hello running home01

Every node answers the same commands. The rest of this ladder types on home01 whether or not it leads.

A node that is being retired leaves on purpose. shc cluster leave on the node refuses while deployments run there, so empty it first: take a2 off far01, then drain it. traefik’s global replica stops in place.

home01
$ shc uninstall a2 --no-backup -y
...
$ shc node drain far01 --yes
...
far01
$ shc cluster leave --yes
...

The leave moves far01’s cluster identity into a dated directory under /var/lib/shc and keeps its images and volumes. The cluster forgets it.

home01
$ shc cluster status
cluster name: shc-cluster
leader: home02
nodes: 3 healthy of 3
voters: 3
quorum: healthy, 2 voters required
leader-only work, recorded by home02 21s ago:
...
home01
$ shc node ls
...
$ shc label ls --key tier
...

Join it again with a fresh token. A node that left joins like a new one, and it comes back active.

far01
$ ssh home01 shc node token --ttl 30m --fields token -o text | shc node join - --name far01 --cloud learn --location far --label tier=edge
...
home01
$ shc node ls --where '@learn/far'
NAME STATE CONDITION AVAIL CLOUD LOCATION ROOM BUILD STORE LEADER VOTER
far01 healthy ready active learn learn/far ok 0.1.47 replica no no
$ shc cluster status
cluster name: shc-cluster
...
nodes: 4 healthy of 4
voters: 3
quorum: healthy, 2 voters required
...

Put a2 back on it.

home01
$ shc install ~/learn/cluster/hello -s a2 --where tier=edge --label owner=learn
...

A node that dies without leaving is removed from another node. shc cluster rm and shc node forget both refuse while a deployment row names the node; the stopped global replica a drain leaves is the one row they allow. The refusal is X400017NODDEP, naming the two rows and the way out. So take a2 off again, drain far01, stop its daemon, then forget it.

home01
$ shc cluster rm far01 -y
far01 is a docker-runtime host.
- 2 deployments still name this node and block its removal until they are relocated or removed; the other docker-runtime nodes are home01, home02, home03
shc: [X400017NODDEP] node far01 is still named by 2 deployment rows: master/traefik, prod/a2; 2 have their deployment directories on this node and SHC has seen their containers here. Relocate them with `shc node drain far01` and clear anything it leaves behind with `shc uninstall <stack>`, then retry, or remove the node anyway with `shc cluster rm far01 --force`
home01
$ shc uninstall a2 --no-backup -y
...
$ shc node drain far01 --yes
...
far01
$ systemctl stop shc
home01
$ shc node ls --where node=far01
NAME STATE CONDITION AVAIL CLOUD LOCATION ROOM BUILD STORE LEADER VOTER
far01 unreachable down drained learn learn/far ok 0.1.47 replica no no
home01
$ shc node forget far01 -y
...
$ shc node ls
...

--rebalance promotes a non-voter inside the control-plane fence before a voter is removed, so the voter set never dips below its target. This cluster has no non-voter inside the fence, so it is not typed here.

explained, not walked: no non-voter inside the fence to promote
$ shc cluster rm home03 --rebalance -y

far01 still holds the identity of the cluster that forgot it. --force on the join wipes that state, and the node joins like a new one.

far01
$ systemctl start shc
far01
$ ssh home01 shc node token --ttl 30m --fields token -o text | shc node join - --name far01 --cloud learn --location far --label tier=edge --force
...
home01
$ shc node ls --where '@learn/far'
NAME STATE CONDITION AVAIL CLOUD LOCATION ROOM BUILD STORE LEADER VOTER
far01 healthy ready active learn learn/far ok 0.1.51 replica no no
home01
$ shc install ~/learn/cluster/hello -s a2 --where tier=edge --label owner=learn
...

shc cluster repair sweeps every node the cluster has confirmed dead; with --force it is the quorum-loss recovery of Disaster recovery, which rewrites raft membership so the node it is typed on is the only voter. --dry-run asks what either form would do and changes nothing. The plain sweep runs from any node. The forced form is for a cluster that has no leader, so on this healthy one its dry run answers the refusal the recovery would give. home01 has been a follower since its restart above, and the refusal names the node that leads.

home01
$ shc cluster repair --dry-run -v
...
$ shc cluster repair --force --dry-run
warning: force repair rewrites raft membership, may lose uncommitted writes, and will split-brain if the other nodes are actually alive
shc: [X409405CLSLDR] refusing force-leader recovery, 10.48.0.2:4002 answers as the cluster's leader, so quorum is not lost and there is nothing to recover

On the leader the same dry run is refused as well, for the voters that still answer, and it names them. Which node leads after the election above is the cluster’s choice, so that one is not typed here.

explained, not walked: the leader after the election is the cluster's choice
$ shc cluster repair --force --dry-run

Keep a1, a2, a3 and a6 for the next page.

--force on cluster leave, cluster rm and node forget skips the deployment-row refusal and leaves those rows naming a node that is gone; --yes on a forced repair confirms the membership rewrite. None belongs on a healthy cluster.

explained, not walked: each would strand rows or rewrite raft membership on a healthy cluster
$ shc cluster leave --force --yes
$ shc cluster rm far01 --force -y
$ shc node forget far01 --force -y
$ shc cluster repair --force --yes

Next: Upgrading the daemon.