SHC Network Fabric
Layer 9. Fabric. Fabric
The shc fabric carries container traffic between nodes. shcd owns its address allocation and reconciles shc0 with the cluster store. Every container is one routed /32, its pin, on a link of its own to the host. No two containers share a link, and no tenant has a bridge.
Sources: CE modules/fabric/types.go, posture.go, chunk.go, pins.go
and subsystems/impl/fabric_reconcile.go.
Fabric selection
Section titled “Fabric selection”The shc fabric is the workload network. shc init with no --fabric sets
it up wherever the node’s runtime can attach it: docker, podman and incus on
Linux, and apple on a Mac. --fabric none is refused there (X400389FABOFF).
Where the runtime cannot attach it, docker or podman on a Mac, the cluster
is formed without a fabric. Init says so in its output and shc doctor
reports it under fabric.posture. --fabric shc on such a node is refused
(X400395FABATT). --fabric swarm is refused (ce modules/fabric/types.go,
X400371FABSWM). --runtime swarm is refused
(ce modules/cluster/service.go, X400422CLSRUN).
The address block
Section titled “The address block”--address-block selects the IPv4 block SHC carves its own ranges from,
chosen at genesis. It is not tied to the fabric mode: mesh addresses and
service VIPs come out of it too, so it is recorded whether the fabric is
shc or none. Unset is 10.64.0.0/16. A block is a /8 at the largest
and a /16 at the smallest, written as its own network address, and
refused (X400349ADRBLK) when it overlaps the docker address pool
172.16.0.0/12.
core/addrblock carves the genesis block by fixed fractions, whatever its
size. The first eighth
is the operator’s own: SHC never addresses it and never checks a network in
it. Service VIPs take the third sixteenth and mesh addresses the fourth.
Deployment networks take the second quarter, a /25 each: a docker, podman
or apple node carves the whole quarter for its own bridges, and the members
of an Incus cluster carve it together, because an Incus bridge is every
member’s. On Incus a network holds one /25 on every member it runs on, and
keeps it through a move, and once the quarter is full the members take
/25s from the blocks the cluster added. The fabric takes the second half,
which nodes take in /26 chunks. On the default block that is these ranges:
10.64.0.0/19the operator’s own networks10.64.32.0/20service VIPs10.64.48.0/20mesh addresses10.64.64.0/18deployment networks10.64.128.0/17the fabric
A site picks a block whose four ranges are clear of every network any node
is on, VPNs included. shc init and shc node join refuse a node with a
network or route inside one of them (X400450ADROVL at init, X409450ADROVL
from the leader at join), naming the interface and the range. A site that
keeps its own LANs low in a /16 can lay the block over them: with its
LANs inside 10.236.0.0/19, --address-block 10.236.0.0/16 leaves them in
the operator’s eighth and gives service VIPs 10.236.32.0/20, mesh
addresses 10.236.48.0/20, deployment networks 10.236.64.0/18 and the
fabric 10.236.128.0/17. Do not use 100.64.0.0/10: carrier-grade NAT and
VPNs live there. shc cluster fabric status prints the block in its own
cell and the four ranges as rows under it; -o json carries them as
block and ranges.
The genesis block keeps its carve for the life of the cluster. A /16
block’s fabric half, a /17, holds 512 chunks of 64 addresses, and the
first is never handed out. Every node holds at least one, so the default
block has room for 511 nodes. A /14 block’s /15 fabric holds 2,048
chunks, and a /8 block’s /9 holds 131,072. A cluster that needs more
room adds a block: shc cluster block add <cidr> records one more IPv4
block, a /8 to a /26, as one pool of /26 chunks. A /24 added block
is four chunks. An added block is not carved. Node
fabric chunks, service VIPs and mesh addresses are drawn from it once the
genesis ranges are full. The add is refused when the block shares an
address with the genesis block, a block already added, the docker pool, or
a network or route on any node, and while any node cannot be asked for its
networks. A block is never removed or shrunk. shc cluster block list
prints the genesis block, then every added block with how many chunks it
holds, how many are held and how many are left. Sources: CE
core/addrblock/addrblock.go, pool.go, modules/fabric/blocks.go and
modules/cluster/block.go.
A docker or podman node’s engine network shc_fabric holds every fabric
range as a subnet, the genesis fabric and each added block, so the engine
takes a pin of any of them. Neither engine adds a subnet to a network that
exists, so an added block reaches a node as shc_fabric made again, one node
at a time. A node whose network lacks a range takes the cluster’s one turn,
a store row that lapses after ten minutes if its holder goes quiet. It stops
every container on the network and every container in one’s network
namespace, takes each off, makes the network again over every range, puts
each back at its pin with the gateway priority and aliases it had, starts
what was running, and gives the turn up once those run and pass their health
checks. A journal under the state home names what it moves before anything
stops, so a daemon that ends part way finishes the move at its next sweep. A
node that is down when the block is added takes its turn when its daemon is
back. shc cluster block add prints each node’s progress and returns once
every node holds the block; shc cluster block list and the doctor check
fabric.network-ranges say where each node stands. Incus attaches a pin as a
routed NIC and apple gives each pin a network of its own, so neither holds a
range and neither takes a turn. ip shc_fabric_nat covers every range; no
rule of SHC’s in iptables names one (SHC-3322). Sources: CE
modules/fabric/network.go, network_roll.go, network_recreate.go and
glue_linux.go.
Runtime picture
Section titled “Runtime picture”A node takes /26 chunks of the fabric as it needs them: one when it
enrols, and one more each time the chunks it holds are full. Its first
chunk’s .1 is the node’s fabric address, which shc0 holds as a /32.
Every other address of its chunks except each chunk’s first and last can
be a pin: 62 of a chunk’s 64 addresses, 61 of the first.
A pin is a /32 from its node’s chunks, whatever its tenant. On docker and
podman the container joins the node’s engine network shc_fabric, whose
driver shc-fabric is SHC’s own: docker’s remote network and IPAM driver,
which shcd serves, and the netavark plugin podman runs. It is no bridge. On
incus the executor turns the attachment into a routed NIC with no parent.
Each way the container’s end carries the pin as a /32, with a route to the
node’s fabric address on the link and the default route through it. The
host’s end is a veth of its own, named shr and the pin’s four bytes in hex
(10.84.7.130 is shr0a540782). It carries the node’s fabric address and
holds the host’s route to the pin. No bridge sits anywhere on the path, and
the tenant holds nothing on the node: it is a label the node’s policy reads
from the pin ledger.
On a Mac shc0 is the daemon’s userspace WireGuard device on a utun, and the
Mac publishes its key only once its pf policy anchor com.apple/shc/policy
is loaded. Apple takes no address it is given, so each pinned replica runs
on a vmnet network of its own, a /30 of the node’s chunks, 14 to a chunk
once the /30s holding its first and last address are left out. Its pin is the
one address vmnet hands out there, and the host’s bridge is the network’s
other end. One network per replica keeps every container on a link of its
own with the host, as a veth does on Linux.
Pins are rows of the store’s pin ledger, fabric_pin. A service keeps its
pin across renders on the same node, and two allocators racing for one
address collapse on the ledger’s unique node and address. A released pin’s
address stays out of allocation for 10 minutes, and every node refuses
traffic to it for the first five, so a client that still holds it fails at
once. A chunk nothing sits in any more goes back. The chunk holding the
node’s address stays, and so does one taken in the last 10 minutes, which
its allocator is about to fill. A given-back chunk, or the chunks of a
removed node, go to another node only after cluster.address_quarantine
(1 h) and once every live node has said its shc0 config no longer names
them.
Why per-node chunks and routed /32s
Section titled “Why per-node chunks and routed /32s”Addresses come in per-node /26 chunks of one shared fabric range because
shc0 routes each peer’s chunks. A peer’s WireGuard AllowedIPs are the chunks
it holds, so the peer state on every node is a few stable /26s that change
only when a node takes or gives back a chunk, never one route per container
rewritten across the cluster. A new container is one pin drawn from its own
node’s chunks and changes no peer’s configuration. WireGuard also drops
every packet whose source is outside the sending peer’s AllowedIPs, so a
node can send only from its own chunks.
A chunk is a /26 so that a block’s fabric is cut fine enough for many
nodes. At /24 the default block’s fabric held 128 chunks, and a cluster
stopped near 128 nodes however few containers each ran. At /26 it holds
512. A node that runs more than 61 containers takes another chunk, which
adds one entry to each peer’s AllowedIPs.
A container’s address is a /32 on a routed NIC so that nothing is
switched. No two containers share a link, so every packet a pin sends or
receives passes the node’s policy on that pin’s own veth, a co-located pair
included. No L2 is shared, so nothing reaches another container except as
a routed packet the policy has judged.
The trade is that an address belongs to its node. A container that moves to another node gets a new pin from the new node’s chunks, and its old address goes into quarantine. The stable handle for a service is its VIP and its DNS name. The node’s DNS answers the new address from the next revision, the VIP dispatches to whichever replica pins are ready, and no connected deployment re-renders.
shc0 and the path between nodes
Section titled “shc0 and the path between nodes”shc0 listens on UDP 4243. It routes the fabric range over the device, and
every block the cluster added beside it, and WireGuard’s AllowedIPs pick
the peer. It dials every peer at its underlay address, never inside mesh0.
It starts at MTU 1280, the least an IPv6 link carries and the least an
Incus NIC takes, and every pin’s veth or routed NIC is made at shc0’s MTU.
A pair of nodes runs a larger per-route MTU only while both ends’ readiness
rows are fresh and a probe has proven the path, bounded by each end’s link
MTU and the proven path MTU, less WireGuard’s 80. The floor needs an
underlay link of 1360: shc init and shc node join refuse a node whose
link carries less (X400390FABMTU).
flowchart LR A["Container on node A, its pin a /32"] --> B["Its own veth on A"] B --> C["shc0 on A"] C -->|"WireGuard over underlay: UDP 4243"| D["shc0 on B"] D --> E["Its own veth on B"] E --> F["Container on node B, its pin a /32"]The nft table ip shc_fabric_nat keeps a fabric-to-fabric flow’s real
source. Its chain hooks postrouting just ahead of the engines’ masquerades
and maps the flow to its own source, so the receiving node sees the
container’s pin. The same chain masquerades a pin’s flow to anything outside
the fabric: a pin’s veth is no engine’s bridge, so no engine’s masquerade
covers it. One fabric-to-fabric flow is masqueraded anyway: a replica that
dials its own service VIP. Once DNATed it runs from its pin to the same pin,
so the postrouting_hairpin chain of ip shc_vip hooks ahead of the
exemption and masquerades it to the node’s address on the replica’s
interface. Sources: CE modules/fabric/wireguard.go, endpoints.go,
natexempt.go and modules/vip/compile/compile.go.
SHC writes no rule into iptables on any engine. A pin is a routed /32 on a
veth of its own, so dockerd’s chains, its masquerade and docker 28’s raw
drops, all keyed to a docker bridge or its subnet, never match one, and the
identity mapping in ip shc_fabric_nat decides a fabric flow’s NAT before
any engine’s masquerade runs. What is left is dockerd’s ip filter FORWARD
policy drop, which takes every packet dockerd did not accept, a pin’s among
them. That is node setup, not a rule SHC fights: shc init and
shc node join put "ip-forward-no-drop": true in /etc/docker/daemon.json
(Docker 28 and newer), checked with dockerd --validate and put back as it
was if dockerd rejects it. The key acts at dockerd’s start, so a drop dockerd
set before it stays until the next boot, or until an operator runs
iptables -P FORWARD ACCEPT. Podman needs nothing in iptables with either
netavark firewall driver: netavark never sets a FORWARD policy, and its
rules, iptables or inet netavark, name its own bridges and subnets, never
a pin’s veth. A daemon upgraded from v0.1.16 takes out what it kept in
iptables (SHC-FABRIC, its FORWARD jump, the nat and raw head rules,
SHC-MESH-CLIENT and its masquerade) on its first fabric pass, and only
those. SHC’s own rules live in its own nft tables, the operator-client
verdict in ip shc_mesh_client among them. A program that drops a bridge’s
own traffic at the forward hook (dockerd before setup, a dockerd beside
Incus, ufw, firewalld) is not patched around. The doctor check
fabric.forward-path probes the path with two throwaway endpoints on a
throwaway bridge, on every engine, and names the program and its fix, and a
job that times out reaching its peer carries the same reading. A node runs
one container engine. Sources: CE modules/fabric/glue_linux.go,
iptables_release_linux.go, meshclient_nft.go, forward_path_doctor.go,
core/fwdhook and core/hostactivate/fabric_host_setup.go.
Host networking: setup, not a sweep
Section titled “Host networking: setup, not a sweep”The host’s networking is shared with every other program on it, so SHC sets its part up once and reports what changes afterwards instead of putting it back (SHC-3323). Every host setting the fabric sweep or the VIP pass used to re-assert, and where each lives now:
| Setting | Kind | Why |
|---|---|---|
net.bridge.bridge-nf-call-iptables = 1, with br_netfilter loaded | setup | the host’s, shared with every engine; without it a frame one bridge switches between two of its ports never meets SHC’s policy table |
net.ipv4.conf.all.accept_source_route = 0 | setup | the host’s; with it on, a destination read at the forward hook is a packet’s first hop, not where it goes |
net.ipv4.ip_forward = 1 | setup | the host’s; every pin flow to another node, a relay and a mesh gateway are forwarded. It was re-asserted by the relay and the mesh-client glue |
dockerd’s "ip-forward-no-drop": true | setup | dockerd’s own configuration; without it dockerd drops forwarded traffic it did not make when it turns forwarding on |
SHC-FABRIC, its FORWARD jump, the nat RETURN, the raw ACCEPT, the pins’ egress and relay accepts | gone | they lived in docker’s tables and were hoisted back to the head every sweep and on every docker network event; nothing needs them on a set-up node |
SHC-MESH-CLIENT and its masquerade in iptables | gone | the same verdict lives in ip shc_mesh_client |
ip shc_fabric_nat, ip shc_mesh_client, ip shc_fabric_loopback, the crossing guard and SHC’s policy tables | sweep | SHC’s own tables, no other program writes them; each follows the cluster’s fabric ranges, blocks, pins and tenants, which change while the node runs |
route_localnet on shc0 and each pin’s veth | sweep | SHC’s own devices; a pin’s veth exists only while its container runs |
| the node’s fabric network on its engine | sweep | SHC’s own network; it is remade when the cluster adds a block |
shc init and shc node join write the kernel settings to
/etc/sysctl.d/90-shc-fabric.conf and br_netfilter to
/etc/modules-load.d/shc-fabric.conf, and apply them to the running
kernel. The daemon applies the kernel settings once when it starts, which
carries a node set up by an older release, and never again in that process.
No watcher runs on a docker or podman node’s networks. When an operator or
another program changes a setting, it stays as they left it, and the doctor
check fabric.host-setup fails naming the setting, the value it found, the
value setup put there and the command that puts it back. It warns when a
drop-in is missing or differs, and on a docker node when daemon.json does
not set ip-forward-no-drop. Sources: CE core/hostnet,
core/hostactivate/fabric_host_setup.go, modules/fabric/host_setup_linux.go
and host_setup_doctor.go.
Compose UX: authored vs rendered
Section titled “Compose UX: authored vs rendered”Apps declare services in compose. SHC translates or refuses deploy:
keys according to Compose keys.
| Rendered element | Fabric behaviour |
|---|---|
| Fabric network | Every pinned service joins shc_fabric as an external network, with its pin as ipv4_address and its default route there (gw_priority) |
| Authored networks | Deployment networks: a /25 of the deployment range each, at MTU 1500, recorded in networks.json. Node-local on docker, podman and apple; one /25 for every member on Incus |
| Service addresses | Pins from the fabric ledger |
| Provider names | Recorded in the deployment’s stored spec; every node’s DNS answers them with the provider’s live addresses |
| Ingress upstream | Service VIP for replica projects; ready fabric pin for a deployment project |
The render joins no consumer to a provider’s own network: a same-tenant
provider is reached at its pin, and Traefik dials a routed service on its
pin. The renderer writes no extra_hosts entries for provider discovery.
The node’s DNS answers from live state. Sources: CE
modules/app/fabric_render.go, modules/depnet/depnet.go,
modules/resolver/source.go, modules/app/ingress_fabric_upstream.go and
ingress_vip_upstream.go.
Lifecycle flows
Section titled “Lifecycle flows”Enrolment gives a node its fabric address and first chunk only when it can attach the fabric. shcd reconciles the device and peers from membership. mesh0 enrolment and raft startup are described in Cluster join.
Install and update draw pins and render fabric attachments. Node removal
tombstones the node’s row, releases every live pin on it and gives its
chunks back; every peer drops them from shc0 on its next reconcile and says
so. Sources: CE modules/fabric/repo.go, chunk.go, pins.go and
modules/app/fabric_render.go.
Reachability control: the policy compiler
Section titled “Reachability control: the policy compiler”The compiler builds per-node nftables rules from deployments and their declared ports. The deployment is the policy unit, and a pin’s tenant is the label it is filed under.
| Traffic | Rule |
|---|---|
| Within a deployment | All ports allowed |
| Within a tenant | Other deployments reach a service only on its declared container ports |
| Across tenants | Blocked except for the derived system roles |
| System ingress | Reaches routed backends on their route ports, and the service an armed tcp_ingress binding names on its declared port |
| To the system ingress | Every tenant’s containers reach its pins on the ports it publishes, keeping their own address, unless a published port’s listener took the connection |
| System collector and shared provider | Reachable on their declared ports |
| Other fabric traffic | Dropped |
A plug supplies connection details; it does not grant permission. A
cross-tenant plug is refused with X400372FABXTN. A service with no
ports declared by ports: or expose: is private to its deployment.
Published ports have separate listener rules, requested by --expose.
The compiler checks same-node traffic as well as traffic through shc0. Two
pins on one node are two veths, so their traffic is routed and meets the
same forward hook. fabric_sources binds each pin to its own veth and the
node’s chunks to its pins’ veths, so a packet that does not wear the pin of
the veth it entered on is dropped, whatever the container can do inside its
own namespace. fabric_entry drops a packet entering a pin’s veth from any
local bridge: a deployment’s own network, docker0, a podman or incus
network. A node whose uplink is itself a Linux bridge (br0, vmbr0, a
bridge over a VLAN) names it to the compiler, so those checks read it as the
network the node faces and not as a container’s bridge. fabric_outside
takes a packet for the node’s chunks from a source outside them that did
not enter on shc0: a host on the uplink, a VPN or a local bridge. It passes
the replies of accepted flows, an ICMP error about a flow a pin here opened,
a listener’s flow to a published port and an incus node’s collector export,
and drops the rest, so a host that routes the pin range through a node
reaches no pin (SHC-3484). Linux’s
inet shc_node_plane table keeps containers off control listeners, with
explicit allowances such as the node DNS resolver. These controls do not
make a node a boundary for hostile tenants; see Security.
Sources: CE modules/fabric/policy/doc.go, policy/compile.go and
policy/hostplane.go.
Node plane and container fabric
Section titled “Node plane and container fabric”mesh0 carries raft and the internal API on node mesh addresses. shc0 carries container traffic. Both tunnels dial the peer’s underlay address with separate WireGuard keys and UDP ports, so neither tunnel runs inside the other. On Linux both are kernel devices. A Mac runs both as the daemon’s userspace WireGuard devices on utuns and joins raft as a non-voter. See Two WireGuard tunnels, sections 2 and 13.
Each device’s key rotates the same way, by shc cluster mesh rotate [<node>][:<device>] and on the leader’s schedule (cluster.key_rotation,
30 days): every peer installs the pending key beside the old one, the node
switches, and a peer moves the node’s addresses on the first handshake under
the new key. shc0’s key is on the node’s fabric grant; a Mac’s is not
rotated. The policy’s forward chain clamps the MSS of every SYN in or out of
shc0 to rt mtu, so a container’s TCP fits the peer’s route without an ICMP
reaching it.
An apple node holds its containers off the node plane with the pf child
anchor com.apple/shc/node-plane, compiled from the same HostInput as
the Linux nft table inet shc_node_plane, and loaded by shcd while the Mac
is a member. Its policy anchor com.apple/shc/policy holds each replica
network’s bridge to its one address, in place of fabric_sources. The
design is in Security, mesh
networking.