Skip to content
SHC Docs

Cordon, drain, undrain, uncordon

A node’s availability says whether it takes work. active takes it; cordoned takes nothing new and keeps what it has; draining is being emptied; drained is empty. Nothing moves by itself: every change below is typed.

Cordon home02. A new install lands elsewhere; a1, a2 and a3 are untouched.

home01
$ shc node cordon home02
warning: node home02 is cordoned; this binds SHC's own placement only: it is excluded from install-time scoring and from drain relocation targets, but a compose deploy typed against it directly with `shc install` on that node or `--node home02` still lands there
✔ cordoned node "home02"; nothing running on it was touched
$ shc node ls
NAME STATE CONDITION AVAIL CLOUD LOCATION ROOM BUILD STORE LEADER VOTER
far01 healthy ready active learn learn/far ok 0.1.47 replica no no
home01 healthy ready active learn learn/home ok 0.1.47 voter yes yes
home02 healthy cordoned cordoned learn learn/home ok 0.1.47 voter no yes
home03 healthy ready active learn learn/home ok 0.1.47 voter no yes
$ shc install ~/learn/cluster/hello -s a6 --where '@learn/home' --label owner=learn
installing deployment "a6"
[a6] Healthy 1/1
✔ nginx healthy
✔ installed deployment "a6"
$ shc ls --where stack=a6
STACK TENANT ENV APP STATUS NODE
a6 prod prod /root/learn/cluster/hello running home03
home01
$ shc node uncordon home02
✔ node "home02" is schedulable again

Drain home03. Each deployment moves to the node its rules allow, one at a time; a3 prefers rack=r1 but any node in learn/home will do. The node ends drained.

home01
$ shc node drain home03 --yes
home03 is a raft voter in slot 2 of 3 and a docker-runtime host.
- draining does NOT change raft membership: home03 stays a configured voter and keeps counting toward quorum, so fault tolerance is unchanged: it is the workload that moves, not the vote
- 2 deployments must relocate; the other docker-runtime nodes are far01, home01, home02
- traefik runs on every node, so its replica here stops
warning: drain of home03 relocated a3, a6 BRING-UP-FIRST: each mounts no volume, so its copy on the target was up and ready before the one on home03 stopped, and it was never down
✔ drained node "home03"
$ shc ls --where owner=learn
STACK TENANT ENV APP STATUS NODE
a1 prod prod /root/learn/cluster/hello running home01
a2 prod prod /root/learn/cluster/hello running far01
a3 prod prod /root/learn/cluster/hello running home02
a6 prod prod /root/learn/cluster/hello running home02
home01
$ shc node ls
NAME STATE CONDITION AVAIL CLOUD LOCATION ROOM BUILD STORE LEADER VOTER
far01 healthy ready active learn learn/far ok 0.1.47 replica no no
home01 healthy ready active learn learn/home ok 0.1.47 voter yes yes
home02 healthy ready active learn learn/home ok 0.1.47 voter no yes
home03 healthy drained drained learn learn/home ok 0.1.47 voter no yes

Uncordon returns the node to service. Nothing moves back. a3 returns when its rule says so.

home01
$ shc node uncordon home03
✔ node "home03" is schedulable again
$ shc update a3 --where node=home03 --yes
relocation plan for prod/a3/prod
placement: home03 is the only node its rules, condition and capacity leave
replicas: nginx r0 moves home02 -> home03
volumes: none
durable data: 0 B to carry
downtime: about 30s
✔ relocated deployment "a3" to home03
$ shc ls --where stack=a3
STACK TENANT ENV APP STATUS NODE
a3 prod prod /root/learn/cluster/hello running home03
$ shc update a3 --where '@learn/home' --prefer rack=r1 --yes
updating deployment "a3"
[a3] Healthy 1/1
✔ nginx healthy
source: /root/learn/cluster/hello, replayed from the deployment's persisted source (path)
the rollout of prod/a3/prod to v1:sha256:78fdce16baee809f341064f02397eb1080475fa44fe37b5be8e18f3d45082456 finished

Undrain stops a drain and leaves the node cordoned. --timeout caps the whole run in seconds.

home01
$ shc node drain home02 --timeout 600 --yes
home02 is a raft voter in slot 3 of 3 and a docker-runtime host.
- draining does NOT change raft membership: home02 stays a configured voter and keeps counting toward quorum, so fault tolerance is unchanged: it is the workload that moves, not the vote
- 1 deployment must relocate; the other docker-runtime nodes are far01, home01, home03
- traefik runs on every node, so its replica here stops
warning: drain of home02 relocated a6 BRING-UP-FIRST: each mounts no volume, so its copy on the target was up and ready before the one on home02 stopped, and it was never down
✔ drained node "home02"
$ shc node undrain home02
✔ the drain of node "home02" is stopped, 1 deployment had moved off it and 1 was stopped in place; the node is cordoned
home01
$ shc node ls
NAME STATE CONDITION AVAIL CLOUD LOCATION ROOM BUILD STORE LEADER VOTER
far01 healthy ready active learn learn/far ok 0.1.47 replica no no
home01 healthy ready active learn learn/home ok 0.1.47 voter yes yes
home02 healthy cordoned cordoned learn learn/home ok 0.1.47 voter no yes
home03 healthy ready active learn learn/home ok 0.1.47 voter no yes
$ shc node uncordon home02
✔ node "home02" is schedulable again

--plan asks where a drain would put each stack and changes nothing. It prints a row per stack with what the drain would do, the node it would go to and what blocks it. Ask it about far01.

home01
$ shc node drain far01 --plan
...

a2 may only run on tier=edge, and far01 is the only such node, so a drain of far01 would have nowhere to put it and would refuse. Move a2 home first. The drain then has only traefik’s global replica, which stops in place; uncordon starts it again, and a2 goes back to the edge.

home01
$ shc update a2 --where '@learn/home' --yes
...
$ shc node drain far01 --yes
...
$ shc node ls
...
$ shc node uncordon far01
✔ node "far01" is schedulable again
$ shc update a2 --where tier=edge --yes
...
$ shc ls --where stack=a2
...

--force stops in place what no other node can take, a deployment pinned to the node included, instead of refusing. a2 is pinned to the edge, so a forced drain of far01 stops it there, and it stays stopped after the uncordon.

home01
$ shc node drain far01 --force --yes
...
$ shc ls --where stack=a2
...
$ shc node uncordon far01
✔ node "far01" is schedulable again
$ shc ls --where stack=a2
...

A maintenance window is a cordon with an announced return: the scheduler does not fail over the node’s work while it is away inside the window. --drain empties it first; --end lifts the window and leaves the node where the drain put it, so uncordon it.

home01
$ shc node maintenance home03 --window 15m
warning: node home03 is cordoned; this binds SHC's own placement only: it is excluded from install-time scoring and from drain relocation targets, but a compose deploy typed against it directly with `shc install` on that node or `--node home03` still lands there
✔ node "home03" is in maintenance until 2026-11-06T17:46:12Z, availability cordoned: nothing is rescheduled from it before then
$ shc node maintenance home03 --end
✔ maintenance on node "home03" lifted, availability active
$ shc node maintenance home03 --window 15m --drain --yes
home03 is a raft voter in slot 2 of 3 and a docker-runtime host.
- draining does NOT change raft membership: home03 stays a configured voter and keeps counting toward quorum, so fault tolerance is unchanged: it is the workload that moves, not the vote
- 2 deployments must relocate; the other docker-runtime nodes are far01, home01, home02
- traefik runs on every node, so its replica here stops
warning: drain of home03 relocated a3, a6 BRING-UP-FIRST: each mounts no volume, so its copy on the target was up and ready before the one on home03 stopped, and it was never down
✔ drained node "home03"
✔ node "home03" is in maintenance until 2026-11-06T17:46:45Z, availability drained: nothing is rescheduled from it before then
$ shc node maintenance home03 --end
warning: node home03 is still drained: `shc node uncordon home03` returns it to service
✔ maintenance on node "home03" lifted, availability drained
$ shc node uncordon home03
✔ node "home03" is schedulable again

Run shc ls once more: a2 is still stopped where the forced drain left it, the others are running, a3 and a6 where the last move put them, and every node is active.

home01
$ shc ls --where owner=learn
STACK TENANT ENV APP STATUS NODE
a1 prod prod /root/learn/cluster/hello running home01
a2 prod prod /root/learn/cluster/hello stopped far01
a3 prod prod /root/learn/cluster/hello running home02
a6 prod prod /root/learn/cluster/hello running home02
home01
$ shc node ls
NAME STATE CONDITION AVAIL CLOUD LOCATION ROOM BUILD STORE LEADER VOTER
far01 healthy ready active learn learn/far ok 0.1.47 replica no no
home01 healthy ready active learn learn/home ok 0.1.47 voter yes yes
home02 healthy ready active learn learn/home ok 0.1.47 voter no yes
home03 healthy ready active learn learn/home ok 0.1.47 voter no yes

Keep a1, a2, a3 and a6 for the next page.

Nothing moves back on its own after a drain. shc rebalance --plan prints the moves that would even out the load, with each one’s explain, and changes nothing; --apply would run them.

home01
$ shc rebalance --plan
...

--apply runs the plan’s moves after a confirmation --yes skips. The next pages count on the deployments staying where these pages put them, so it is not typed.

explained, not walked: the next pages count on where the deployments are
$ shc rebalance --apply --yes

Next: Nodes failing and rejoining.