The mesh under it all
Two WireGuard tunnels run between every pair of nodes. mesh0 is the node
plane: the store’s replication, the internal API and the node DNS travel
on it. shc0 is the fabric from the last page, where containers travel.
Both dial the peer’s underlay address, each with its own key and UDP port,
so neither runs inside the other. shc cluster mesh status is the node
plane as this node sees it, one row per node with its mesh address and
whether its tunnel is installed and running:
$ shc cluster mesh statusmesh: upNODE NAME MESH ADDRESS IS LOCAL INSTALLED RUNNING NODE ID...shc cluster mesh typed with no subcommand prints the same status:
$ shc cluster meshEach node’s mesh address is in the mesh range, 10.192.0.0/11 on the
default address block, handed out in join order: home01 is 10.192.0.1,
far01 10.192.0.2. shc node ls -v shows it in the INTERNAL column,
with the port the internal API answers on, beside the underlay address
the tunnel dials:
$ shc node ls -vNAME ID HOST INTERNAL UNDERLAY RUNTIME CORES MEMORY FREE CORES FREE MEMORY OUTSIDE RESERVED STATE CONDITION AVAIL CLOUD LOCATION ROOM BUILD LEADER VOTERfar01 node-fc10df0b46b8 far01 10.192.0.2:4003 65.21.55.73 docker 2 7.6G 2.0 7.6G - - healthy ready active learn learn/far ok 0.1.2 no nohome01 node-666d6f4dc640 home01 10.192.0.1:4003 167.233.221.15 docker 2 7.6G 2.0 7.6G - - healthy ready active learn learn/home ok 0.1.2 yes yesThe devices on the node, and the ports they listen on:
$ ip -4 addr show mesh07: mesh0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000 inet 10.192.0.1/11 brd 10.223.255.255 scope global mesh0 valid_lft forever preferred_lft forever$ ip -4 addr show shc0... inet 10.128.16.1/32 scope global shc0...$ ss -lunState Recv-Q Send-Q Local Address:Port Peer Address:PortUNCONN 0 0 10.192.0.1:53 0.0.0.0:*...UNCONN 0 0 0.0.0.0:4243 0.0.0.0:*UNCONN 0 0 0.0.0.0:4244 0.0.0.0:*...4244 is mesh0, 4243 is shc0, and port 53 on the mesh address is the
node DNS. shc cluster mesh diagnose has this node probe every peer over
the mesh and report the round trip:
$ shc cluster mesh diagnoseFROM MESH ADDRESS TUN PEER PEER IP REACHABLE RTT MS...The node DNS answers the cluster’s private names: every node under
nodes.shc.internal, every service under its tenant. Ask far01’s
daemon for home01:
$ dig +short @10.192.0.2 home01.nodes.shc.internal10.192.0.1A node’s mesh0 and shc0 keys can be rotated while the cluster runs. Each
rotation is prepared on its node, every peer installs the new key beside the
old one, and then the node switches; shc cluster mesh status shows it until
it settles. With nothing named, this node’s mesh0 key rotates and then its
shc0 key.
$ shc cluster mesh rotate✔ rotation active: node-666d6f4dc640 mesh0 epoch 1✔ rotation active: node-666d6f4dc640 shc0 epoch 1A node’s name rotates that node’s keys from here, and a device after a colon rotates that one alone:
$ shc cluster mesh rotate far01:shc0✔ rotation active: node-fc10df0b46b8 shc0 epoch 1The leader also rotates every key once it is 30 days old, one key an hour,
and cluster.key_rotation sets the age, or turns the schedule off:
$ shc config set cluster.key_rotation 7dRun the status command again until the rotation line is gone, then probe the peers once more.
$ shc cluster mesh statusmesh: upNODE NAME MESH ADDRESS IS LOCAL INSTALLED RUNNING NODE ID...$ shc cluster mesh diagnoseFROM MESH ADDRESS TUN PEER PEER IP REACHABLE RTT MS...Every peer reaches home01 on its new key. shc cluster mesh client
enrols an operator’s Mac on the mesh the same way. The cluster lists the
clients it has:
$ shc cluster mesh client lsno clients in tenant "prod"The rest runs on the Mac: enrol it under a name, bring its mesh0 up,
read or move the gateway it dials, and revoke it when it goes:
$ shc cluster mesh client enrol --name laptop$ shc cluster mesh client up$ shc cluster mesh client peers --name laptop --current home01$ shc cluster mesh client revoke laptop --yesThe cluster keeps everything this ladder installed: web with two
replicas, hello, site and www with their routes and records, dns
and the connection.
Next: Your own ingress.