Skip to content
SHC Docs

Two WireGuard tunnels, never double wrapped (design)

Status: open design (SHC-1201). mesh0 and shc0 are on next, and the leader signs the peer table (SHC-1286); a follower that answers an enrol asks the leader to sign the joiner’s table (mesh.enrol.sign, SHC-1668), which section 7 does not describe yet. Key rotation is built for both devices (SHC-1345, lane MESHROT): shc cluster mesh rotate [<node>][:<device>] runs section 8’s prepare, acknowledge, switch and activate, a peer keeps the old key until the rotating node has switched (SHC-1815), and the leader rotates each key past cluster.key_rotation on a schedule. A Mac’s shc0 key does not rotate. Relay and punching (sections 6 and 11, SHC-1285) are built, shc0 takes each peer’s route MTU from its own probe (SHC-1284), and a Mac node runs shc0 with a fabric grant of its own once its pf policy loads (SHC-1623, SHC-1642). Not built: section 4’s ICMP confirmation under a temporary nftables rule and section 5’s mesh.mtu.fragmented counter; confirmation on next is handshake-based. Clay accepted the target on 2026-09-30 (“ok 2 tunnels but not double wrapped”). On next (the new line) the node plane is mesh0 and Nebula is deleted (stage 2c, SHC-1201). main and release/v0.0 keep Nebula because the live estates run them, and the bitspur move is app by app onto new nodes. Stage 1 was ce !270 (tests in shc !513); slice 2 made it the only path and its switch is deleted (section 13).

Out of scope, by ruling: any service mesh or L7 proxy (closed, no-service-mesh). This is layer 3 only: kernel WireGuard, routes, bridges and nftables. There is no per-packet code of ours and no new tunnel protocol.

A node runs two overlays today, one inside the other:

  • Nebula (nebula0, UDP 4242, tun MTU 1300) carries the node plane. Since SHC-1101 the node row’s internal_address is the node’s mesh address, so rqlite and the internal API (:4003) ride Nebula. Enrol and the mesh’s own repair calls dial underlay_address (ce cluster.UnderlayAddrFor).
  • shc0 (kernel WireGuard, UDP 4243) carries containers. ce subsystems/impl/fabric_reconcile.go dials every peer at its internal_address, which is now a Nebula address. So every cross-node container packet is WireGuard inside Nebula, and fabric.MTU is 1300 - 80 = 1220 for shc0 and every fabric bridge.

On the rig (section 10) the double wrap moves 0.72 to 0.96 Gbit/s container to container at MTU 1220, with Nebula busy about one core on each side. Kernel WireGuard dialling the underlay directly moves 4.0 to 4.5 Gbit/s at MTU 1420 to 1440 on the same two VMs.

Two kernel WireGuard devices per node:

mesh0shc0
carriesnode plane: rqlite (raft and HTTP), internal API, VIPscontainer traffic
addressone /32 per node from the control-plane range 10.192.0.0/10the node’s fabric grant .1, carried with the fabric prefix
AllowedIPs per peerthe peer’s /32, plus VIPs it holdsthe peer’s fabric grant
UDP listen port42444243 (unchanged)
keyits ownits own (already in fabric_allocation.public_key)

The ports are fixed per device and the same on every node. mesh0 never takes Nebula’s 4242, so deleting Nebula at stage 2c closes a port and moves nothing. Neither device uses 443: UDP 443 stays with Traefik, which may serve HTTP/3 later (not ruled). Enrol needs only TCP 443 outbound from the joiner (section 7) and opens no inbound port for itself. The tunnels then need UDP 4243 and 4244 between nodes, which a node behind NAT gets through its observed endpoints (section 11).

The fabric range is 10.128.0.0/10 with a per-node grant that is a genesis knob defaulting to /20, and node-local deployment networks come from 10.64.0.0/10 (ruled 2026-10-01). The rig keeps its 10.84.0.0/16 until it is rebuilt; nothing here renumbers a live cluster.

For a given peer both devices start from the same underlay address, so each packet is wrapped once. They are still two UDP flows with different ports: each roams on its own, can get its own NAT mapping and can hash onto its own ECMP path. Everything that is learned about a path (observed endpoint, confirmation, MTU) is therefore kept per device, and a result on mesh0 says nothing about shc0.

Keeping containers off the node plane. WireGuard’s cryptokey routing drops a packet whose source is not in the sending peer’s AllowedIPs, so no shc0 peer can claim a 10.192.0.0/10 source. That does not stop a legitimate fabric source addressed to a node-plane destination. The enforcement is nftables, in the per-node ruleset SHC-1202 compiles (ce modules/fabric/policy):

  • forward: drop anything from a container-facing interface (shc0, the fabric bridges, deployment bridges, container veths) to mesh0 or to 10.192.0.0/10, and anything from mesh0 to a container-facing interface;
  • input: a packet arriving on a container-facing interface reaches only the host services declared for containers (the per-node resolver and the local public listener); the internal API, rqlite and every other host port drop. One narrow exception on shc0 itself: ICMP echo request and reply between a peer grant’s .1 and this node’s .1 (the confirmation and black-hole probes of sections 4 and 5), and the ICMP errors path MTU needs (fragmentation needed, packet too big, time exceeded);
  • input on mesh0: accept only from peer mesh addresses that are in the current table, the same ICMP set included.

The device split makes those rules interface matches rather than address lists, which is what keeps them short and auditable.

shcd is the coordinator: it owns the peer table, endpoint ranking, keys, routes and probes, and drives both devices over netlink exactly as modules/fabric already drives shc0 (wireguard.go, applier_linux.go).

3. The peer table, on the foundation ce already has

Section titled “3. The peer table, on the foundation ce already has”

Nothing below is a new mechanism. The store, node liveness, request/response to one node and fire-and-forget events are built in ce (memory shc-foundation-already-built): core/db over rqlite, modules/nodelease, modules/internal_rpc, modules/events. The mesh is rows plus a reconcile loop, like the fabric.

Rows. Most of a peer is already stored:

  • node: id, internal_address, underlay_address, public_address, mesh_address, location, provider.
  • fabric_allocation: the fabric grant, shc0’s public key, tombstones that keep a grant from ever being reused.

Stage 2a adds three tables, all in the single squashed initial migration (new line, no compat):

mesh_peer, keyed by node id, the identity:

columnmeaning
node_idPK, the node row
kindnode or client (section 9)
mesh_public_keymesh0’s public key
mesh_addressthe /32 in 10.192.0.0/10, allocated at enrol, tombstoned forever like a fabric grant
key_epochmesh0’s active key epoch
updated_at

mesh_rotation, keyed by (node id, device), a key rotation in progress on one device (section 8): pending_public_key, epoch, prepared_at, node_ready (the rotating node’s own acknowledgement). A device’s active key lives where it always does: mesh_peer.mesh_public_key for mesh0, fabric_allocation.public_key for shc0.

mesh_endpoint, one row per (observed node, device, observer), the paths:

columnmeaning
node_idthe node this endpoint reaches
devicemesh0 or shc0
observerthe node that saw it, or the node itself for an address it reports
kindlan, public or observed
ip, portthe endpoint
locationthe observer’s location, for LAN ranking
observed_atrefreshed while it stays true; rows older than 10 minutes are ignored

Each node writes only rows where it is the observer, as conditional updates, and only on change or refresh. Liveness is not duplicated: a peer whose node lease has expired is still a peer (its tunnel may come back), and ranking reads the lease’s condition to try it last.

The signed table. Identity is what a node must not take from an unauthenticated source, so the leader signs it. The signed object is a snapshot of both devices’ membership: every mesh_peer row and every live fabric_allocation row (grant and shc0 public key), every open mesh_rotation row (node, device, epoch, pending public key), plus each node’s identity key (below), with a generation (bumped on every join, rotation and revocation), a content digest, issued_at and not_after = issued_at + 7 days. It is signed with the cluster signing key, the key every join token pins (section 7). That is everything a node needs to bring both devices up before rqlite is reachable. The leader re-signs at least hourly and on every generation bump.

A node replaces its snapshot with one of a higher generation, or with one of the same generation and digest and a later issued_at (a refresh that only extends not_after). The same generation with a different digest is a leader fault: the node keeps what it has and raises a doctor finding.

Identity keys are not WireGuard keys. WireGuard keys are Curve25519 Diffie-Hellman keys and are never used to sign. A node signs requests with its existing cluster identity key, the key behind its internal-API client certificate (ce modules/ca), which the snapshot binds to its node id. It is rotated by the CA’s own procedure, never by a mesh key rotation, so a node can always authenticate the recovery path in section 7 even halfway through rotating its tunnel keys.

Endpoints are not signed: a wrong endpoint cannot impersonate a peer, because WireGuard authenticates every packet by key, and a node can only advertise endpoints as itself.

Change propagation. Reconcile is the source of truth, as for the fabric (a node that was down converges when it returns, with no catch-up protocol). A sweep runs every minute on every node and compares the store’s generation, the snapshot and the kernel. To make a change usually take effect at once rather than at the next tick, the writer also publishes mesh.peer.changed through modules/events, which broadcasts to every peer with per-peer FIFO and recovery on start; a receiving node runs its sweep early. A lost event costs up to one sweep interval, never correctness.

Reads that must not need the leader. rqlite’s default read consistency goes through the leader, so the mesh read path asks for local reads explicitly (read consistency none) and accepts their staleness. Each node also keeps the newest signed snapshot it has seen, from the store or from a peer (section 7), and never replaces it with a lower generation.

4. Candidates, ranking and confirmation, per device

Section titled “4. Candidates, ranking and confirmation, per device”

Gathering. For each device a node reports:

  1. LAN addresses: every global-unicast address on an interface that is not SHC’s own (mesh0, shc0, fabric and deployment bridges, nebula0 while it exists, container veths), with the device’s port.
  2. The configured public_address, when the operator set one.

and records what it observes of others: Linux WireGuard sets a peer’s endpoint to the outer source of each handshake or transport packet it accepts from that peer, in processing order (after authentication and the replay check), unless shcd sets it over netlink. So outside shcd’s own writes, the endpoint node B’s shc0 holds for A is A’s shc0 flow as seen from B, address and port. B writes it as an observed row (node A, device shc0, observer B). No STUN server is needed; the peers are the reflectors, per device.

Ranking, per peer and per device, highest first:

  1. Same site and a LAN candidate in a subnet this node also has: LAN direct. Two nodes share a site when their location paths agree on the cloud and the place, the first two segments (gorillaservers/ogden/incus01 and .../incus02 do; hetzner/fsn1 and hetzner/hel1 do not). A same-site dial no handshake proves within five minutes is dropped for the next candidate, and tried again after an hour (SHC-2768).
  2. Public direct: the peer’s public_address, or a fresh observed endpoint for that device seen by a node in this node’s location.
  3. Relay through a public node (section 6).

LAN-direct is decided per pair: home01 reaches home02 over the LAN and far01 over the public path at the same time.

Confirmation is an authenticated round trip, not a handshake. Setting a new endpoint does not force a fresh handshake (an established session keeps being used), so a handshake timestamp proves nothing about the new path, and sampling the endpoint before and after a trial cannot prove it did not roam away and back in between. A trial is therefore coordinated: the end that wants it asks the other (mesh.trial, internal RPC over whatever path works now) for an instant two seconds ahead and a trial window, and at that instant both set exactly the endpoints they hold for each other, the candidate pair, for its duration. Coordination alone does not stop a packet already in flight, a NAT rebinding, or a tuple that used to belong to another peer from arriving on another tuple, so for the window each end admits only the candidate tuple to that device’s port: a temporary nftables rule in prerouting drops every other UDP packet to the port, from any peer. Any packet the device accepts during the window, from the peer under trial or anyone else, came from the candidate tuple.

That briefly cuts the device’s other peers, so the window is kept short: the three probes go out together at its start, and it lasts twice the pair’s measured RTT plus 50 ms, capped at 400 ms. Their packets that drop are retransmitted by their own transports (TCP, raft), and 400 ms is well inside rqlite’s one-second heartbeat and election timeouts. For the same reason a trial runs only when something changed: this node set a new endpoint for the peer, WireGuard roamed it, or a kept endpoint failed (its handshakes stopped, or a trial of it failed). There is no periodic re-trial of an endpoint that keeps working: its handshakes are the evidence that it still does, and a re-trial of every endpoint every 10 minutes would drop a device’s other peers for about 20 seconds in every 10 minutes on a 50-node cluster. A node runs one trial per device at a time, leaves at least 2 seconds between trials so timeouts measured since last contact recover in between, and tries one peer at most every 30 seconds. An endpoint the kernel already holds when shcd starts is kept on its next handshake, since nothing changed, and so is any endpoint while no trial can run: a node without nftables (a Mac), a pair that is relayed, or a peer that does not answer the trial. Container UDP that drops in a window is lost, not retransmitted, which is the price of the attribution.

The probes are ICMP echoes carrying a fresh random identifier to the peer’s address on that device (the peer’s mesh /32 on mesh0, its grant’s .1 on shc0); the peer, under the same rule at its end, answers them. A reply carrying the identifier that arrives inside the window entered this end over the candidate tuple, and the probe it answers entered the peer’s end over the candidate tuple at that end. The trial confirms the candidate when both ends report a matched probe and reply inside the window. The rule is removed when the window closes, success or not. A packet in flight from another tuple is dropped; a NAT rebinding during the window makes the trial fail, which is the right answer, and the next trial runs on fresh state.

When the peer cannot be reached by any path, a trial cannot be coordinated: it then confirms only that some path works, and shcd keeps whatever endpoint WireGuard roams to after the reply, recorded as observed, until a coordinated trial can rank it. On failure shcd moves down the list. A confirmed endpoint is sticky until confirmation fails three times in a row, so a flaky higher candidate never causes flapping. When WireGuard roams on its own, shcd reads the new endpoint back, records it as observed, and confirms it before keeping it.

5. MTU, per peer and per device, from the outer path

Section titled “5. MTU, per peer and per device, from the outer path”

WireGuard never probes, and a tunnel MTU that is too large fails badly: on an IPv4 underlay Linux WireGuard sends its outer packets with DF clear, so an oversized inner packet is fragmented on the outer path and is either slow or, where a middlebox drops fragments, a black hole while small pings still work. For the same reason an in-tunnel ping with DF set proves only the inner limit: it can succeed through outer fragmentation. The probe must therefore measure the outer path, not the tunnel.

shcd borrows RFC 8899’s search rules (a base that is assumed to work, raise only on proof, black-hole detection, periodic re-probe) but probes outside the tunnel, from the host:

  • What it sends: ICMP echo with DF set (IPv4) or an unfragmentable echo (IPv6), from this node’s underlay address to the peer’s confirmed endpoint address for that device. These are host packets on the underlay route, so the tunnel’s route and device MTU do not limit them; the search can go above whatever is installed.
  • Search: start at the floor, an outer size of 1280 (the IPv6 minimum), and search upward to the egress link MTU. A size counts as proven after two replies; an unanswered size is not retried upward in the same round.
  • Result: in-tunnel MTU = proven outer size less 60 (IPv4 underlay) or 80 (IPv6), written as a per-route MTU (below).
  • When the floor fails (ICMP filtered, a NAT router answering for the peer, or a path below 1280): the result is “unproven” and the peer stays at the floor in-tunnel MTU, 1200. That floor is only safe on a path that carries 1280 bytes, which every IPv6 path must and every path we run does; an IPv4 path below 1280 is unsupported. A peer whose floor in-tunnel probe also fails is reported unreachable on that device and ranking moves to the next candidate or to relay.
  • Re-probe every 10 minutes and whenever the device’s endpoint for that peer changes.
  • Black-hole detector: shcd sends one in-tunnel probe at the installed size and one at the floor every minute per peer; when the full-size one fails twice while the floor answers, that peer’s route drops to the floor at once and the search restarts.

The MTU is therefore an estimate, and the design says so rather than claiming a proof. ICMP can hash onto another ECMP path than the UDP flow, a NAT router can answer for the peer, and on IPv4 Linux WireGuard cannot be made to set DF on its outer packets, so outer fragmentation cannot be prevented, only observed. Three detectors watch the estimate on the actual traffic:

  1. Fragments that arrive. Each node counts, in an nftables prerouting chain at a priority before defragmentation, first fragments (MF set, offset 0, which still carry the UDP header) from each peer’s endpoint to each device’s port. A count means that peer’s packets toward this node are too big. The receiver tells the sender directly, mesh.mtu.fragmented(device, count) by internal RPC over mesh0 or the 443 path, never through the store, and the sender drops that route to the floor and restarts the search.
  2. Fragments that are dropped upstream never reach the counter; the black-hole detector above sees the full-size probes fail while the floor answers.
  3. Fragments reassembled upstream (a middlebox) are invisible to both, but then traffic is delivered, so the cost is performance, not loss.

A path every detector is blind to is one that delivers. A path that fragments at the floor, or fails the floor, is reported unsupported on that device. IPv6 outer packets are never fragmented in flight, so on IPv6 the black-hole detector alone covers it.

The result is written as a per-route MTU: <peer grant> dev shc0 mtu N and <peer /32> dev mesh0 mtu N. Linux honours a route’s explicit MTU for forwarded packets (ip_dst_mtu_maybe_forward), which is what container traffic is; no sysctl is needed. Each device’s own MTU is the largest of its routes, and the fabric bridges are created at shc0’s MTU. The policy compiler adds an MSS clamp on shc0 (tcp option maxseg size set rt mtu) so TCP never depends on ICMP reaching a container: the forward chain of SHC’s own ip shc_fabric table opens with it for every SYN in or out of shc0, and the kernel takes rt mtu from the smaller of the route each way, which on shc0 is the peer’s probed route. A Mac’s pf has no rt mtu and clamps nothing yet.

shc0 has its own probe on next (SHC-1284, ce subsystems/impl/fabric_reconcile.go): each peer’s route starts at the floor and is raised only on a proven outer path MTU. Refusals and the readiness row keep the egress link’s MTU.

When a peer is reachable by neither LAN nor public direct, shcd routes the pair through a public node R by moving AllowedIPs, not by adding a protocol. Relay still needs UDP from both ends to R; it covers NAT and missing routes, not a blocked UDP.

A relayed pair (A, C) via R on one device is decided by the pair’s canonical owner (below) and applied by both ends, because cryptokey routing checks sources on receive:

  • on A: C’s prefixes on that device (C’s /32 on mesh0, C’s grant on shc0) leave A’s peer entry for C and join A’s entry for R;
  • on C: A’s prefixes leave C’s entry for A and join C’s entry for R. Without this C drops A’s packets arriving from R, because their source is in A’s AllowedIPs, not R’s;
  • on R: nothing changes in WireGuard (R already has A and C as peers with their own prefixes); nftables on R accepts forwarding between those two peers’ prefixes on that device, for pairs R is the current relay for, and nothing else.

R must have a direct confirmed path to both ends on that device; a relay is never itself relayed, which is what keeps R’s own configuration unchanged and rules out routing loops.

One prefix has exactly one peer owner per device. shcd moves a prefix with one wgctrl configuration request that removes it from one entry and adds it to the other, but the kernel may apply a request across several messages, so the prefix can be briefly owned by neither entry (its packets drop for that instant). A request that fails part way leaves the device in some mix of old and new; the next sweep re-applies the full desired state, which is idempotent. Return routing needs nothing extra: each end’s kernel route to the other’s prefix is the connected device route, and WireGuard picks R by AllowedIPs.

Who decides, without quorum. A relay change must work exactly when a path has just failed, possibly with quorum gone, so it does not go through the store. Each pair has a canonical owner, the end with the lower node id; only the owner decides. It picks R, the lowest node id among nodes that publish a public address, have a confirmed direct path to it (a handshake within ObservedHandshake) and are not relayed themselves; ranking by round-trip time is not built. It sends mesh.relay.set(device, a, c, relay, generation) by internal RPC to R over its direct tunnel, carrying the owner’s signed proof; R checks that it has a direct confirmed path to the other end too, forwards the call, and returns the other end’s acknowledgement.

Generations are per pair and per device. The owner keeps the last one it issued on disk; each end keeps the last one it applied on disk. An end applies a strictly higher generation, re-acknowledges an identical one without reapplying it (so a lost acknowledgement is recovered by retrying the same call), and ignores a lower one. Nothing goes to the store: every generation lives in <state home>/mesh/relay.json (the owner’s issued ones, each end’s applied ones, the pairs a relay forwards). A node that restarts reads its own file, keeps the relays it had applied, and the owner’s next check confirms or replaces them.

Failover. Detection runs on the mesh sweep, once a minute and on a kick. A direct path counts as failed on a sweep only once its last handshake is older than ObservedHandshake (3 minutes), or, for a pair that never handshook, once this end has dialled it that long; three failed sweeps in a row declare it dead. A path that dies is relayed about five to six minutes after its last handshake. Changing the relay or going back to direct is then the same call with the next generation; until both ends have applied it, packets in one direction are dropped. The owner gives a candidate three attempts with a 5-second timeout each. If it does not acknowledge, the pair is reported down on that device and the next sweep’s decision starts from the next candidate; a pair with no candidate is reported down as well. While a pair is relayed the owner re-sends the applied generation to its relay every sweep; three failed checks drop that relay from the candidates and the pair is decided again, and three sweeps in a row with a fresh direct handshake send it back to direct. Each end re-confirms the pair after applying.

A release (relay "") follows owner → current relay → other end, with the owner’s signed proof unchanged. The other end applies and acknowledges through the relay before the owner applies; only an owner with no applied relay sends a release directly. If the relay’s own entry has no handshake within ObservedHandshake for three consecutive sweeps and the pair’s direct entry has a fresh handshake, each end drops the relay locally without RPC, persists a local fallback at its held generation, and warns once. A higher owner generation supersedes that fallback; the owner decides again from fresh health.

What R sees: node-plane traffic is TLS end to end (mTLS on :4003, rqlite’s own TLS), so R relays ciphertext. Container traffic is plaintext at R, a node of the same cluster that is already trusted with a tenant’s workloads when they land there.

Raft rides mesh0, so rqlite cannot bring the mesh up. Each node keeps a local cache, /var/lib/shc/mesh/cache.json (root, 0600), rewritten after every successful sweep: the newest signed snapshot, the endpoint rows, and per peer and device the endpoint that last confirmed.

Boot order.

  1. shcd starts, reads the cache, verifies the snapshot’s signature against the cluster signing key pinned at enrol, and checks not_after. It brings up mesh0 and shc0 with the last confirmed endpoints before rqlited starts.
  2. rqlited binds the mesh address and finds its peers.
  3. The first sweep against the store replaces the cache.

When the cache is not enough. If no cached endpoint confirms within 30 seconds, or the snapshot is past not_after, the node fetches a fresh snapshot over HTTPS 443 through Traefik into the public API, the same route enrol uses (built by SHC-1207 behind the one RPC dispatch). It resolves the estate’s host name, whose A/AAAA records the dns module already publishes for the public nodes, and asks any of them. The request is authenticated by the node’s identity key (section 3), a signature over a fresh server nonce, checked against the key the snapshot binds to that node id, so it keeps working while the node’s tunnel keys rotate. The response is the signed snapshot plus the serving node’s endpoint rows. Trust never rests on TLS: the node verifies the snapshot’s signature with the pinned cluster key and refuses a lower generation than it holds. A node answering this path serves the newest snapshot it holds from its own cache, so the path works while quorum is lost and before rqlited is up anywhere. There is no lighthouse role, and the internal listener binds the mesh address only, never the underlay.

Enrol is the same route with a single-use, short-TTL join token instead of a node signature. The token pins the cluster signing key; the response (the new node’s grant, mesh address, the snapshot and the leader’s signature over the new row) is signed, so the joiner verifies it before trusting anything. The route is rate-limited per source and forward-auth lets exactly that path through.

Freshness and revocation. A snapshot is a statement of membership at a generation, not forever:

  • a node applies a higher generation as soon as it sees one (event, minute sweep, any 443 fetch) and drops every peer the new snapshot no longer lists;
  • a node holding a snapshot past not_after keeps its existing tunnels up (the data plane is not torn down because the leader is gone) but accepts no peer it does not already have, raises a doctor finding and an alert, and polls the 443 path every minute;
  • local reads can be stale, and a healthy tunnel never triggers the 443 path, so every sweep also reconciles with peers: as part of the minute’s confirmation round a node asks each peer it has a direct mesh0 path to for its snapshot generation (one small RPC per peer) and fetches the snapshot from the first one that is higher;
  • a revocation therefore reaches a node within one sweep (usually at once, through the event) after the newer snapshot reaches any node it has a direct mesh0 path to, or the store copy it reads. A node with no such path gets it when it next reaches the 443 path. The 7-day validity bounds how long a node can add peers from a stale table, not how long it keeps running;
  • a cached peer answers but quorum is unreachable: tunnels come up from the cache, the node takes the highest generation any reachable peer serves, and joins, rotations and revocations wait for quorum. Nothing else does.

Each device has its own static key. WireGuard already rotates session keys every two minutes, so static rotation is for containing a compromise, not for forward secrecy. WireGuard cannot hold two keys for one peer with the same AllowedIPs, so a rotation is break-before-make per pair; the design bounds the break and gives every miss a way back. On next the break on mesh0 is one handshake (SHC-1815): while a rotation is open every peer holds a second entry for the rotating node, under its pending key and with no AllowedIPs, and moves the node’s prefixes to it only once the node has switched. No peer drops the old key first.

Rotation (shc cluster mesh rotate [<node>][:<device>], and on a schedule), one node and one device at a time. With no device named, mesh0 rotates and then shc0. The leader’s mesh_key_rotation cron starts at most one rotation an hour, the oldest key past cluster.key_rotation (30d), each key dated by a kv row written with its activation or the first tick that saw it. A Mac’s shc0 key does not rotate: the Mac withdraws and republishes it with its pf policy. Every step is keyed by (node, device, epoch) and is idempotent, so any lost request or response is recovered by sending it again:

  1. Prepare. The node generates the new key pair for that device and writes the new private half to disk (root, 0600) beside the old one before anything else, so a crash cannot leave the cluster expecting a key the node no longer has. Over internal RPC it asks the leader to record a mesh_rotation row (node, device, the new public key, epoch = active epoch + 1), retrying every 10 seconds. The leader writes the row, bumps the snapshot generation and publishes mesh.peer.changed. A retry with the same key and epoch is answered as success.
  2. Acknowledge. The prepared rotation is in the signed snapshot, so every peer that applies it adds the pending entry to its device and, once the device holds it, answers mesh.rotation.ack(node, device, epoch, digest of the pending public key), naming exactly what it acknowledges, stored as a row; an acknowledgement for another key or epoch counts for nothing. The entry under the current key is not touched, so the acknowledgement rides the link it leaves alone. The rotating node acknowledges too (node_ready), and only once it has read its own prepared row back.
  3. Switch. When the rotating node and every node with a live node lease have acknowledged, voters and non-voters alike, the leader records the rotation as switching, signs that, and hands the table to the rotating node. The node brings its device up under the new key and then answers, naming the key its device runs. A non-voter is waited for too: one that does not hold the pending key at the switch is cut off from the rotating node, and with one voter that is its only way to the store. A node whose lease is not ready holds nothing up, and neither does a non-voter that mesh0 no longer counts. A node that has not acknowledged after 2 minutes is named by shc cluster mesh rotate (X504816MSHACK) and in shc cluster mesh status; nothing switches, and at 5 minutes the rotation is dropped like one without node_ready, the old key still active everywhere. A switching rotation cannot be aborted. Each peer looks at its pending entry four times a second, and on a completed handshake moves the node’s AllowedIPs to it and drops the old entry in one write. The node asks every peer for its table right after it switches, so each of them sees that handshake at once.
  4. Activate. On the rotating node’s answer, and only on an answer that names the pending key, the leader writes the new key into the device’s home (mesh_peer or fabric_allocation), deletes the rotation row, bumps the generation and delivers the table to every node with a live lease. shc cluster mesh rotate answers rotation active once that delivery is over. A prepare whose response was lost and that the node then abandoned (mesh.rotation.abort(node, device, epoch), or no node_ready within 5 minutes) expires, and the old key stays active. The rotation of a node that left mesh0 is dropped in any phase.
  5. Recover. A peer that saw no handshake switches when it applies the activating generation: on the event, at its next sweep, at the peer reconciliation of section 7, or, when its confirmation of the other end fails, by fetching the snapshot over the 443 path (which does not ride mesh0; 10-second timeout, retried every 30 seconds). The rotating node signs that request with its identity key, not a tunnel key, so the recovery path accepts it in the middle of the rotation. Expected outage, not a bound: a pair that saw the handshake is down for that handshake and at most a quarter second more. When one end missed it, the pair is down until that end notices (its next sweep or a failed confirmation, within about 60 seconds), finds a node serving the current generation, and completes a handshake. Discovery asks the public nodes in turn, 10 seconds each, a stale answer counting as a failure, and repeats the round every 30 seconds; with k public nodes reachable and one of them current, that is at most k times 10 seconds per round, plus WireGuard’s own handshake retries (5 seconds apart). If no reachable node serves the current generation, the pair stays down until one does, and doctor names it. After activation the node drops the old private half at the next successful sweep.

Revocation is deleting the row (and tombstoning its addresses) at a new generation: every peer drops the entry on the event, the next sweep or the next 443 fetch (section 7). This replaces Nebula’s blocklist and certificate expiry.

The darwin shcd embeds wireguard-go (golang.zx2c4.com/wireguard) in process on utun devices and runs as a launchd system daemon. The control code is the same; only the device seam differs (netlink on Linux, the wireguard-go device and PF_ROUTE routes with -mtu on darwin).

  • A Mac node (the apple runtime, SHC-1200) runs mesh0 from stage 2b, so it is a full member of the node plane. It runs shc0 and takes a fabric grant only once SHC-1200 settles how the apple runtime attaches containers to a bridge (open, section 15). Until then a Mac node’s containers are not on the fabric and the node holds no grant.
  • The shc CLI on an operator’s Mac joins mesh0 as a client (stage 2d): kind = client in mesh_peer, a /32 from the client block 10.223.0.0/16 (the mesh pool’s last /16), no fabric grant, no shc0, no containers. The operator enrols it through the public API as a logged-in platform admin (shc cluster mesh client enrol): the CLI makes its own key, posts only the public half, and gets back its address and its gateways. There is no join token. An operator’s Mac already holds admin credentials, so a token would add a public entry point and protect nothing more (SHC-1288). revoke tombstones the row the way a node’s removal does, and ls lists clients. The client peers only with gateway nodes, the nodes with a public address. A gateway’s entry for the client allows exactly the client’s /32; the client’s entry for its current gateway allows the mesh pool. sudo shc cluster mesh client up holds the client’s utun in the foreground and refreshes its gateways every minute. Through a gateway a client reaches the public API listener (TCP 8000) on node mesh addresses and nothing else, masqueraded to the gateway’s mesh address, so no other node needs a route back to client addresses and every prefix keeps a single owner. The verdict is stateless, in mangle PREROUTING ahead of DNAT: the internal API, rqlite, every other port and every other client are dropped, and a node that is not a gateway drops all client traffic. A client never calls internal RPC and holds no certificate. It replaces NetBird for the Mac to cluster path only.

iperf3 between two containers, one on home01 and one on home02. Both are VMs on the same Proxmox host (pve02, 4 vCPU each, Debian 13, kernel 6.12), so the “wire” is a host bridge, not a network. Throwaway devices, addresses and ports beside the production Nebula, all removed afterwards; evidence and the change ledger are in ~/.drover/state/evidence/SHC-1201/measure/. Throughput is the sender’s rate from iperf3. Cores are each VM’s whole busy CPU over the run (iperf3 included), sender / receiver.

pathcontainer MTU1 stream4 streamsreverseRTT avgcores, 1 stream (send / recv)cores per Gbit, 1 / 4 streams
A. WireGuard inside Nebula (today)1220721 Mbit/s9509600.63 ms1.74 / 2.566.0 / 4.9
B. Nebula routes only, grant as unsafe_routes130010449898780.45 ms1.57 / 2.083.5 / 4.0
B’. same at Nebula’s largest tun MTU1440116099511160.44 ms1.65 / 2.133.3 / 3.9
C. WireGuard direct on the underlay14204035418040980.39 ms1.26 / 2.220.86 / 0.92
C’. same at the IPv4 MTU1440446443720.43 ms1.37 / 2.270.82 / 0.86

Host to host for reference: 50 Gbit/s on the shared bridge, 1.26 Gbit/s over the production Nebula. Path MTU on the underlay was 1500 (DF 1472 passed, 1473 failed); container MTUs above were each verified with DF pings at the limit.

What it shows, for these two VMs: Nebula’s userspace forwarder topped out near 1.1 to 1.3 Gbit/s at about 1.4 cores per side whatever the MTU. Removing the inner WireGuard (B against A) gained 45 percent on one stream, 4 percent on four, and lost 9 percent in reverse, all under that ceiling. Kernel WireGuard on the underlay (C) was about four times faster and spent under one core per Gbit across both ends. What it does not show: a ceiling for other CPUs, real NICs or a WAN path, or bitspur’s workload capacity; it was one box with no wire. It is still enough to rank the options: Nebula-direct needs grants signed into certificates for a gain the two-tunnel design beats by a wide margin, so it is not worth building as an interim. It also measured UDP 4245 between the underlay addresses on the rig; that stage 1’s UDP 4243 is open on a given estate is a precondition to check there, not something this proves.

For the sequencing: the cutover rides path A, roughly 0.7 to 1 Gbit/s between two nodes’ containers on VMs like these. The workloads that would press on that (replicated volumes, bulk copies between nodes) are themselves after the cutover. Stage 1 is one config flip with no Nebula change, so it is the cheap lever if cross-node container throughput becomes a problem after the cutover.

tiercoverswhen
0. staticevery node has a public address or a same-site LAN path; the house reaches in through a pfSense port forwardnow (every SHC node today)
1. observedendpoint-independent NAT with a stable port (pfSense static-port outbound)stage 2a (section 4 learns observed endpoints per device)
2. punchingaddress-dependent NAT: while a pair is relayed and its direct handshake is not confirmed, the owner sends mesh.punch through the relay, which stamps both ends’ observed endpoints; both ends set them on the pair’s direct entry at one instant 2 seconds ahead, keepalive 25 s holds the mapping, and the usual three fresh handshakes release the relaystage 2e, built (SHC-1285)
3. relaypairs with no direct path, symmetric NAT on both ends; still needs UDP to the relaystage 2e, built (SHC-1285, section 6)

Tiers 2 and 3 are on next (SHC-1285); nothing in stage 1 or 2a to 2c needs them for the estates we run.

  • Leader down or quorum lost. Tunnels run from the local tables and cache; only joins, rotations and revocations wait for quorum.
  • Every cached endpoint dead (a node moved, a site renumbered). The 443 path (section 7).
  • MTU black hole on a path. The detector drops that peer’s route to the floor; other peers keep theirs.
  • Clock stepped backwards. WireGuard rejects handshake initiations whose TAI64N timestamp is older than the last one it accepted from that key, so a node whose clock jumped back cannot handshake until its clock passes the old value. A rotation (a new key) clears it; doctor flags the skew.
  • UDP blocked on the underlay. Before it enrols, the joiner tries a WireGuard handshake from a throwaway userspace device (wireguard-go on netstack) to each member the token names, on the ports of the devices it will run: both for a Linux node, mesh0 only for a Mac node without a fabric grant. The member holds the throwaway key as a peer of its live device, with no allowed IPs, for at most 15 seconds, armed by underlay.probe.arm on the 443 join route with the bootstrap token, checked and not spent. A handshake with any member passes the port. Otherwise the member’s counters for the key give the direction: nothing received refuses with X409598UDPOUT (from this node to the cluster), received and answered with X409599UDPBCK (from the cluster back to this node), each naming the port. The refusal comes before the token is spent. When no member can hold the key (an older member, a Mac member whose devices are userspace), the join enrols and the node publishes V300565UDPSKP with the reason once its daemon runs. Doctor’s underlay.udp runs the same check, armed on the same route under the node’s identity, for every peer with no handshake in the last three minutes. Relay does not help here.
  • A relay node dies. Confirmation fails on both ends of each relayed pair and the next generation picks another relay or a direct path (section 6).
  • A rotation half applied. Section 8, step 4.
  • A node returns under a new key while another voter is down. Until rotation and the signed table land (SHC-1345 on SHC-1286), a peer that was away cannot learn that key without the store, which rides mesh0, so the pair stays down until raft answers on that peer again, and doctor (mesh.mesh0) names the peer and what it waits for.
  • Key compromise. Rotate the node, or revoke its row.

Stage 1 (ce !270, tests shc !513) put this behind a cluster.fabric_endpoints switch while Nebula carried the node plane. Slice 2 made a node’s internal_address its mesh0 address, so dialling it would wrap every container packet twice, and the switch is deleted (ce !458, tests shc !718). There is one path:

  • shc0 dials every peer at its underlay address, the host mesh0 dials too. A peer with no underlay address is left out of shc0;
  • each node publishes an underlay readiness row in the kv table (namespace fabric, key underlay, scoped to its node id, value its own underlay link MTU less 80 and the peers its link cannot carry 1220 toward, with a 5-minute expiry the sweep refreshes at half life);
  • a pair runs above 1220 only while both rows are fresh and neither refuses the other. The per-route MTU is the smallest of the two published values and what this node’s shc0 prober has proven the outer path toward the peer carries (SHC-1284): a peer starts at the floor and is raised only on proof. Every other peer runs at 1220, which every path carries;
  • a failed readiness read keeps the last good read until each row expires.

Both ends dial the same underlay hosts, so WireGuard roaming has no second path to move a pair onto, and the settle, leave and roaming machinery stage 1 needed is gone with the switch.

Every intermediate state is safe. The applier creates a fresh shc0 at 1220. On each sweep it lowers first: the device MTU and every owned route whose MTU goes down or goes away change before the WireGuard endpoints change; then the endpoints; then every MTU that goes up. An error at any step returns with the lower value in place, which every path can carry. Owned per-peer routes carry rt protocol 167, are added exclusively and deleted only with that protocol, so a route an operator or another daemon owns is never replaced or removed; a destination someone else holds pins shc0 at 1220 and the sweep reports it. It needs UDP 4243 reachable between underlay addresses.

The bridges. A tenant bridge is created at shc0’s MTU when that is above 1220. Docker and podman cannot change a network’s MTU in place, so a bridge at the wrong MTU is dropped and recreated only on a drained node no replica is pinned to and no container, running or stopped, names; until then its containers keep the MTU they were created at.

Five steps, each its own MR, each behind the previous one being proven.

  • 2a. mesh0 beside Nebula (Linux nodes). The mesh_peer, mesh_rotation and mesh_endpoint tables, the signed snapshot, per-device candidates, ranking, confirmation, the outer-path probe, the cache and the 443 fetch. mesh0 comes up on 4244 with its 10.192.0.0/10 address and carries nothing but probes and diagnostics (shc cluster mesh diagnose grows a mesh0 column). shc0 takes its per-route MTU from its own probe instead of stage 1’s link MTU less 80. Rollback: turn the mesh0 switch off; shcd deletes mesh0 and its routes, and shc0 returns to section 13’s rule (link MTU less 80 between peers whose readiness rows are both fresh, 1220 otherwise) on the next sweep, through the same lower-first ordering.
  • 2b. the node plane moves to mesh0, and Mac nodes join it. New clusters do it at genesis: the genesis node’s internal_address is its mesh0 address and rqlited binds it. An existing new-line cluster moves node by node: drain, leave, re-enrol onto mesh0, the same verbs the Nebula join uses today. A Mac node runs mesh0 from here. Rollback: re-enrol that node onto Nebula, which is still installed.
  • 2c. delete Nebula. modules/nebula, the nebula_mesh table, the lighthouse and Nebula certificate verbs, and the bundled binary go; UDP 4242 closes. mesh0 stays on 4244, so no port moves. shc cluster mesh keeps its name and gains rotate. Rollback: none in place. Roll back the release on fresh nodes and restore, the same procedure as the cutover.
  • 2d. the operator client. Client enrol and revocation, gateways and their masquerade (section 9). Rollback: revoke the client rows; nodes are unaffected.
  • 2e. NAT tiers 2 and 3. Punching and relay (sections 6 and 11), built on next (SHC-1285). Rollback: turn relay selection off; pairs fall back to direct candidates, which is all tier 0 needs.
  • Relay choice by round-trip time, and declaring a dead direct path in seconds rather than over three one-minute sweeps (section 6): not built, not ruled.
  • Whether Traefik serves HTTP/3 on UDP 443 (Clay asked, not ruled). Nothing here depends on it.