Nebula for nodes, the fabric for containers, NetBird for people (2026-09-29)
Superseded by Cluster overview. The text below is the record as of its date.
Status: OWNER-RULED 2026-09-29, measured on a throwaway cell (SHC-1049). The target it belongs to is the multi-site design, SHC-1050 §2.2 and §3.10. This record pins which network does which job, why NetBird is not the cluster network, and what the SHC-1049 cell measured about the mesh as it ships today (shc 0.0.85).
Owner ruling, verbatim: “Ok honestly let’s keep nebula then, but I still use netbird for users that access the private network.”
| layer | carries | on it | boot dependency |
|---|---|---|---|
Nebula (nebula0, 10.42.0.0/16, UDP 4242) | node to node: raft, the rqlite API, the internal API and fabric peers (the target; see finding 1 for today) | every node of a multi-node cluster | none. Cert, CA and static host map are files on the node, and nebula.service starts from them without the daemon, the vault or a leader. |
SHC fabric (shc0, WireGuard) | container to container across nodes | containers | Nebula: fabric peers dial mesh addresses (the target; today they dial internal_address) |
| NetBird, plus the pfSense VPN | people and devices reaching the private network | laptops, phones, admin boxes | nothing in the cluster boot path depends on it |
A one-node estate has no peer to address and needs no underlay. NetBird for people applies to it all the same.
Why NetBird is not the node network
Section titled “Why NetBird is not the node network”A NetBird peer that reboots asks the management server for its peer list before it can reach any peer. The free edition runs one management instance; active-active HA is a paid tier. Hosted on the estate, a full-estate power cut has to bring the management server back before any node can reach another, and losing the node it runs on stalls every join. Hosted off the estate, the cluster network depends on something outside the cluster. Nebula has neither problem: trust is a CA-signed cert on each node, and there is more than one lighthouse (every node today, at least two in the SHC-1050 target), so no one node is needed for the others to find each other.
Revisit only if the constraint changes: a paid, highly available NetBird control plane, or one that lives off the estate by design. Why NetBird’s management API faces the internet is a separate ruling, netbird stays public, and it stands.
Why the fabric stays its own layer
Section titled “Why the fabric stays its own layer”Nebula gives each node one address. The fabric gives each node a container subnet, pinned service addresses and nftables policy (SHC-1055, SHC-1056). It dials mesh addresses, so it rides Nebula, and its MTU drops from 1420 to 1220 (Nebula’s 1300 less WireGuard’s 80).
What the SHC-1049 cell measured
Section titled “What the SHC-1049 cell measured”Three Debian 13 VMs (2 vCPU, 2.5 GB) on pve01. nebula-test-1 and -2 share
one bridge and play the LAN estate. -3 sits on its own bridge, routed
through the hypervisor with 150 ms of tc netem delay, and plays a far node.
The cell was built without --nebula, then shc cluster mesh enable ran,
then the far node joined: the shape Clay asked about, an existing LAN cluster
growing a far node. Evidence, timestamps and raw output are on SHC-1049.
- Nothing SHC dials rides Nebula today. Every raft, rqlite, internal-API
and Swarm session on the cell ran between physical addresses; none had a
10.42.xend. With only UDP 4242 allowed between the far node and the LAN pair, the far node reportedX503021CLSQRM(1 of 3 voters reachable) while its mesh pings answered at 150 ms. It came back healthy about 50 s after the path reopened, Swarm about 2.5 min. So an existing LAN cluster can grow a far node today only over a routed path that carries TCP 4001-4003 and the Swarm ports.mesh enableadds Nebula beside that path and nothing uses it. SHC-1050 §3.10 is the change that makes the mesh the path. - A far node can hold the raft leadership. The far node joined as the third voter and later won the election, so every write paid the WAN round trip. The control-plane fence in SHC-1050 keeps voters, and so the leader, in one site.
- Joining does not extend the mesh (fixed by SHC-1078). Every join
needed
shc cluster mesh enableagain, which rewrites and restarts nebula on every node. The cluster now converges membership itself. - Leaving does not stop Nebula or revoke the cert (fixed by SHC-1078).
After
shc node leave, the removed node still pinged every mesh peer and its cert stayed valid until the mesh CA was rotated. Leave now stops nebula and deletes the cert, and every survivor blocklists it. - The static host map was incomplete (fixed in shc !357). Every node
is a lighthouse and nebula never queries another lighthouse from one, so
a node dials only the peers its own map names.
mesh enablegave the enabling node only itself and each peer only the nodes installed before it. After any restart, reboot or cold boot the enabling node and the early peers could not reach later nodes until those dialled first. With the fix, a full cold boot and a reboot of the CA node both answered in all six directions on the first ping. - The shipped config is slow on a long link (tuned in the same MR). The receiver dropped UDP at the kernel-default 208 KiB socket buffer and the 500-packet tun queue dropped on send. Numbers below.
- The mesh CA key lived only on the enabling node’s disk (fixed by
SHC-1099). Rebooting that node was harmless. Losing it for good meant no
node could sign a new member without minting a new CA. The CA key is
now sealed in the vault, its record and cert ledger are the replicated
nebula_meshrow, and any voter signs. nebulaandnebula-certare not in the deb. The cell installed v1.9.5 from GitHub, as the terraform cloud-init does.- The vault cold-start capability has one holder. When the three nodes lost power together and the holder did not come back, the other two had raft quorum but the vault stayed locked on every node until the holder returned. On the cell the holder failed because the test itself had replaced its binary 30 s before the power cut, not because of an SHC defect; the lock-out is the design.
Recovery, scenario by scenario
Section titled “Recovery, scenario by scenario”| scenario | cluster (raft, vault, Swarm) | mesh, shipped mesh enable | mesh, complete static map |
|---|---|---|---|
| (a) stop each node’s nebula in turn | never degraded: nothing SHC dials uses the mesh | the survivors kept their tunnels; the restarted node could not reach later peers until they dialled it | every direction on the first ping (map written by hand, byte-identical to what mesh enable now writes) |
| (b) reboot the raft leader, then the CA / cold-start holder | healthy again in about a minute, vault unlocked by peer push or the drop-in | the rebooted node could not reach later peers until they dialled it | all six directions on the first ping (mesh enable from !357) |
(c) node leave, destroy the VM, join a fresh one | healthy, voter again; Swarm kept a phantom manager seat for the old node | needed a manual mesh enable; the removed node kept mesh access | as shipped (both are findings 3 and 4) |
| (d) power-cut all three | all healthy 60-80 s after power-on, no manual step | three of six directions failed until the far side dialled | all six directions on the first ping (mesh enable from !357) |
MTU and throughput
Section titled “MTU and throughput”iperf3, 10 s on the LAN pair and 15 s on the far pair. The LAN raw path is
a memory copy inside one hypervisor and is not a NIC figure. Nebula on the
LAN is CPU bound at 2 vCPU.
| path | RTT | tun MTU (largest DF ping payload) | 1 stream | 4 streams |
|---|---|---|---|---|
| LAN, raw | 0.10 ms | 1500 (1472) | 33.8 Gbit/s | 42.5 Gbit/s |
| LAN, Nebula as shipped | 0.31 ms | 1300 (1272) | 1250 Mbit/s | 1209 Mbit/s |
| LAN, Nebula tuned | 0.31 ms | 1300 (1272) | 1190 Mbit/s | 1131 Mbit/s |
| far, raw routed | 150.2 ms | 1500 (1472) | 184 Mbit/s | 760 Mbit/s |
| far, Nebula as shipped | 150.7 ms | 1300 (1272) | 20 Mbit/s | 33 Mbit/s |
| far, 16 MiB socket buffers | 150.7 ms | 1300 (1272) | 99 Mbit/s | 171 Mbit/s |
far, buffers and tx_queue: 5000 (as shipped since !357) | 150.7 ms | 1300 (1272) | 147 Mbit/s | 575 Mbit/s |
far, as above with mtu: 1440 | 150.7 ms | 1440 (1412) | 150 Mbit/s | 576 Mbit/s |
far, as above with routines: 2 | 150.7 ms | 1300 (1272) | 146 Mbit/s | 586 Mbit/s |
nebula sets the socket buffers with SO_RCVBUFFORCE, and the unit grants
CAP_NET_ADMIN, so no sysctl is needed. mtu: 1440 fits a clean 1500-byte
path but gains nothing here, and 1300 leaves room for PPPoE and tunnels on the
way, so it stays. routines: 2 gains nothing on 2 vCPU.