shc node
Cluster Operations
Section titled “Cluster Operations”shc node token
Section titled “shc node token”Generate a join token for adding new nodes to the cluster:
shc node token# Output: eyJ...base64-encoded-token...The token includes:
- Leader address
- Bootstrap address list
- CA certificate
- Encrypted join credentials
--ttl sets how long the token stays valid (default 1h).
shc node join
Section titled “shc node join”Join an existing cluster using a token:
shc node join eyJ...token...Options:
| Flag | Description |
|---|---|
--nic | Network interface for ALL cluster-internal traffic: enroll address, raft (rqlited), internal API bind, mesh0 underlay (auto-detected by default) |
NIC selection (auto-detect):
- Deterministic: candidates sorted by interface name
- Private (RFC1918) addresses preferred over public
- Within the preferred class, the default-route interface wins
- A bridge the node’s container runtime made (
docker0,br-…,podman0,incusbr0), SHC’s own devices and a link that carries no signal are passed over while any other interface is up - Use
--nicto override auto-detection (a nic that fails to resolve aborts the join)
# Auto-detect nicshc node join eyJ...token...
# Pin the cluster-internal nic explicitlyshc node join eyJ...token... --nic eth1On join, the node:
- Receives cluster CA and node certificate
- Joins the raft cluster (as voter or non-voter)
- Starts SHC server with mTLS-authenticated internal API
On a mesh-enabled cluster the new node joins the mesh by itself: the node holding the mesh CA signs its certificate and extends every node’s static host map within about 30 seconds of the join.
shc node leave
Section titled “shc node leave”Gracefully leave the cluster:
shc node leaveThis:
- Checks and drains workloads
- Syncs pending writes
- Notifies other nodes
- On a mesh cluster, takes mesh0 down; the leader then removes its peer from every other node
- Converts back to SQLite
shc node status
Section titled “shc node status”View cluster status:
shc node statusShows:
- Cluster mode (single/cluster)
- Leader node
- All node states
- Quorum status
Node Management
Section titled “Node Management”shc node ls
Section titled “shc node ls”List nodes:
shc node lsshc node ls -v # Verbose output with extra detailsshc node ls --output jsonshc node info
Section titled “shc node info”View node details:
shc node info node01Shows:
- Node ID and address
- Role (leader/voter/non-voter)
- Health status
- Resource usage (CPU, memory, disk)
shc node health
Section titled “shc node health”Check the health of a specific node:
shc node health node01Shows:
- Node connectivity status
- Resource usage
- Service health
shc node drain
Section titled “shc node drain”shc node drain <node> [--force]shc node drain <node> --planDrain stops new placement on a node and relocates its workloads. It asks the
placement engine where each deployment goes, with the drained node excluded:
the target matches the deployment’s --where and the constraints over it, and
the scorer ranks the peers that are left. A volume’s current reach does not bind the
target, because the relocation carries the volume (a backup and restore when
it cannot reattach there). A store’s reach does bind.
Deployments move one at a time, highest scheduler.priority first and the
platform’s own stacks in their recovery order. A global replica is never
moved: it stops last, once everything else has left the node.
A deployment whose --where, or a constraint over it, holds it to the source
node with a node= term is pinned there (ce modules/fence/node_fence.go,
NodeFence): drain refuses before changing state (X400401CLSDRP). A row the
engine cannot place (its --where empties its eligible set, or no peer is
ready) is a failed row that names why; --force stops it in place instead.
Two gates stop the run before it changes anything. A deployment
with a stored group of two or more members is highly available. Drain protects
its last serving copy (X400402CLSDRB). A deployment whose deploy.disruption
is never blocks the drain, which names it (X400423CLSDRM), and a setting
that could not be read refuses too (X503425CLSDBU). A rolling deployment,
the default, moves one replica of a service at a time, each ready on its new
node before the next is touched. --force explicitly accepts the outage. Its journal
records committed and unresolved moves so retries do not repeat a cutover, and
partial results retain their per-stack detail.
--plan asks the same question and changes nothing. It prints one row per
stack: STACK, ACTION (move, stop, stay or blocked), TARGET (the node it would
go to) and BLOCKER (why it can’t move, in the words a drain would refuse
with). When a gate would refuse the whole drain, the summary says so and the
refusal follows the rows. It exits 0 whether or not a row is blocked. Each stack is placed against the cluster as
it is now, while a drain reads it again after every move, and the move’s own
admission isn’t asked, so a planned move can still be refused for cpu, memory
or start room.
shc node undrain
Section titled “shc node undrain”shc node undrain <node>Restores SHC availability to active and closes the open drain attempt.
It does not move workloads back or restart deployments stopped by a forced drain.
shc node forget
Section titled “shc node forget”Remove an unreachable node:
shc node forget node01Use when a node cannot gracefully leave.
A removal that would break raft quorum refuses with X409201CLSRMQ or
X409202CLSRMR, including when --force is set. Removing the final node
tears down the cluster.
Initialization
Section titled “Initialization”shc init
Section titled “shc init”Initialize a new SHC cluster on the first node:
shc initOptions:
| Flag | Description |
|---|---|
--nic | Underlay network interface (auto-detected by default); raft uses mesh0 |
NIC selection (auto-detect):
- Deterministic: candidates sorted by interface name
- Private (RFC1918) addresses preferred over public
- Within the preferred class, the default-route interface wins
- A bridge the node’s container runtime made (
docker0,br-…,podman0,incusbr0), SHC’s own devices and a link that carries no signal are passed over while any other interface is up - Use
--nicto override auto-detection
# Auto-detect nicshc init
# Pin the cluster-internal nic explicitly (e.g., for dual-homed servers)shc init --nic eth1On init, SHC:
- Generates cluster CA and node certificate
- Initializes the state database
- Starts SHC server with internal API
- Installs system tenant apps (Traefik, OpenObserve)
See Also
Section titled “See Also”- Cluster Commands - Cluster health and diagnostics
- Mesh Architecture - Service mesh overview
- Node Management