Skip to content
SHC Docs

Flow: cluster join

Layer 1. Mesh. Mesh

A joining node receives its mesh0 grant during enrolment, brings up mesh0, then joins raft over its mesh address. Mesh0 uses UDP 4244, with kernel WireGuard on Linux or userspace WireGuard on a utun on macOS. A Mac joins as a non-voter, running the rqlited its archive carries at lib/shc/bin/rqlited. The container fabric uses shc0 on UDP 4243, a userspace device on a utun on a Mac on the apple runtime. Both tunnels dial underlay addresses, so each packet is wrapped once.

Sources of truth:

  • SHC modules/node/commands/local_join.go, modules/cluster/enroll_rpc.go, modules/cluster/enroll_handler.go and modules/cluster/join_client.go.
  • CE modules/mesh, modules/cluster/mesh0_orchestrator.go and subsystems/impl/mesh_reconcile.go.
  • Two WireGuard tunnels, sections 2, 3, 7 and 13.

Run shc node token on a healthy cluster member. The daemon records the bootstrap token in the single-use mesh_token ledger. The token carries an expiry, cluster identity, join addresses and the cluster CA trust bundle that pins the key used to verify the signed enrol response.

shc node join <token> decodes and expiry-checks the token. The NIC selection (--nic when supplied) determines the physical address reported as the node’s underlay. It does not select the raft bind address.

The joiner generates its own mesh0 WireGuard key pair and sends only the public key. The private key stays on the node. Cluster formation is guarded by CE core/hostplat: join requires mesh (X501392CLJPLT without it). Control-plane membership requires voter; an explicit fence or label without that capability refuses with X501395ROLCAP.

The joiner calls POST /api/join/node.enroll over HTTPS 443 through Traefik into the public API. The bootstrap token authenticates the request. The joiner verifies the response’s signature against the trust bundle in its token before accepting certificates, grants or peer addresses.

A member that is restarting is not there for some seconds, and a join typed right behind another one meets exactly that: the first join restarts the leader for its storage engine, and every join recreates the edge on each member. The joiner keeps trying every endpoint of its token for 90 seconds while one of them refuses the connection, serves a certificate that does not name its address, accepts the connection and gives no answer, answers 502, 503 or 504, or answers 429, in which case it waits as long as the join route asks. A coded 503 from the handler is tried again the same way: a member with no snapshot signer yet, or with a locked vault, answers with that cause alone, after giving the token, the node row and the mesh0 row back. Any other coded answer is the cluster’s refusal and ends the join at once. When the window ends the join fails with X500023JNFAIL and says the leader was still starting or restarting, with each endpoint’s last answer.

The leader’s enrol handler:

  1. Validates the request, including the mesh0 public key, and consumes the bootstrap token. Failed enrolment releases the consumed token for retry.
  2. Creates or adopts the node identity and records underlay_address from the reported physical IP.
  3. Allocates the mesh0 grant under that public key. The mesh_peer row records the node’s key and address; the node’s internal_address becomes its mesh address on port 4003. Failure to grant mesh0 fails enrolment.
  4. Issues the separate internal-API mTLS certificate, including the mesh address in its SANs, and returns it with the mesh0 grant, fabric posture and raft join addresses on the mesh.

The joiner validates the returned identity, certificate material, mesh0 grant and raft addresses before replacing local identity during --force. It prepares the grant, persists the internal-API certificates and node id, then activates the cached grant and writes the join marker. It brings up mesh0 from that cache and records the bootstrap state before daemon startup.

mesh0 comes up before the daemon’s raft engine. rqlited binds and advertises the mesh address on TCP 4001/4002; the internal API uses the mesh address on TCP 4003. The joiner inherits the leader’s raft TLS posture. The leader decides its raft role from host capabilities and control-plane membership; a Mac is a non-voter.

Identity on mesh0 is the WireGuard public key in mesh_peer. The leader signs the peer table (CE modules/mesh/snapshot_sign.go, SHC-1286), and shcd reconciles the device and local cache against membership. The mesh and fabric reconcile loops each sweep once per minute. The two devices use separate keys and underlay endpoints.

shc cluster mesh status reports the local device and its peers’ handshake health. shc cluster mesh diagnose probes each peer’s internal API over mesh0 from the node serving the command (CE modules/cluster/mesh0_orchestrator.go).

shc node leave refuses a node that still hosts deployments unless forced. It writes its mesh_peer tombstone while it can still reach raft. Peers retain its tunnel while raft still counts it. After raft removal is confirmed, local mesh0 and its cache are removed; the local WireGuard key remains for a rejoin. Survivors drop the departed peer on their next sweep. The tombstoned row retains the address so it cannot be allocated to another node.

A force removal of a node already dropped from raft tombstones its mesh row and asks surviving peers to sweep immediately. Sources: SHC modules/cluster/leave.go; CE modules/mesh/remove.go and repo.go.