Skip to content
SHC Docs

Multi-site estates and fences (design)

Status: open design (SHC-1050). Fences are built and admission reads them. Node labels have replaced operator-made fences (SHC-1355, finished): the operator writes shc node label, shc label, shc constraint and shc selector with --where and --prefer, and shc fence only refuses, naming them. So on next §3.1’s custom origin is label, §3.2’s node.provider is the cloud column, §3.17’s explain is shc constraint explain -s <deployment>, and §3.18’s shc fence and shc rule verbs, shc move --group and the terraform shc_fence and shc_rule don’t exist (terraform has shc_constraint), and --rule and --fence only refuse. Every recommendation this document made was approved, and the owner added decisions of his own (Swarm retired, Nebula always on, fabric for container traffic, no backwards compatibility, fences and rules). They are listed in the decisions. Scheduler mechanics (rescheduling, replicas and spread, failure handling, taints and tolerations) are a separate design, SHC-1058. Moving the bitspur estate onto the new line is SHC-1061; promanager stays on its current line until the owner decides otherwise.

Later the same day the owner refined four points on SHC-1058, and this document follows them: Traefik and the private resolver run on every node, fed from the replicated control plane; a stack renders as one compose project per node it touches, with names resolved live instead of baked in; every Swarm compose key is honoured or refused; and a node that comes back starts nothing until the control plane confirms what it still owns. The four questions that left open were then decided by agent on the owner’s standing delegation, reversibly (decisions 24 to 28).

Written against shc corp/main ce4b456f9 and ce corp/main 3225a49d, and updated against shc 3af98e0f2 and ce 68926d62. Every statement about today’s behaviour cites the file it was read from; ce paths are in the ce repo, everything else is in shc. Where a doc and the code disagree, the code wins and the doc is listed under stale docs.

A word on vocabulary. The ticket says “region”. This document says site in prose for “a place nodes live”, and it deliberately has no built-in region, zone or metro. The only places SHC knows by itself are a node, its provider and the provider’s own location token. Everything coarser is a fence the operator defines, with whatever kind name suits the estate. The examples use custom kinds (site, metro, compliance) on purpose.


The owner wants one SHC cluster to span sites and providers, starting with Hetzner hel1 plus a GorillaServers Proxmox VM, instead of a separate cluster per site. One control plane, one console, one set of tenants. The operator says where things may run; SHC’s one placement engine enforces what physics forbids, keeps chatty integrations together and explains every choice. Nodes run Docker Compose or Podman as executors, Nebula addresses every node, the fabric carries container traffic, and every private name resolves and routes from everywhere.

The end state, as checks a stranger can run:

  1. shc install nextcloud --rule fence=site:ogden lands nextcloud inside site:ogden, and its postgres binding either lands next to it or is refused by name with the command that would satisfy it.
  2. A deployment’s block volume comes from the volume connection whose reach covers the node it lands on. Two volume connections stop being an ambiguity.
  3. shc move inside a volume’s reach rebinds; outside it, copies. A move that leaves the deployment’s fence is refused by name. A move that breaks an integration’s colocation is refused by name and offers the group move.
  4. shc node drain and volume failover pick targets only inside each deployment’s eligible set, and never move voter duty out of the control-plane fence.
  5. Every private hostname resolves from any node, container and NetBird device to a mesh address reachable from all of them, from the resolver on the asker’s own node (a NetBird device asks a node’s).
  6. Every public hostname resolves to the edge of the location the engine placed the app in, and follows the app when it moves.
  7. Cut the WAN between sites: the home site keeps full function, the far site keeps serving what it runs, and nothing split-brains. Today that does not hold for far-site writers on SHC-managed volumes, which stop themselves (5.8); decision 24 exempts far-location volumes.
  8. shc fence explain <deployment> says why it runs where it runs, or why a placement was refused, naming every fence involved.

Out of scope: active-active apps across sites, a database stretched across sites, automatic cross-site failover of stateful apps, a multi-master control plane, per-site tenancy, and backwards compatibility with today’s clusters (6.3).


What exists today and what this design builds on or replaces:

piecestate todaywhere
node locationnode.location holds the provider’s location token, set at self-register from SHC_NODE_LOCATION or the cloud metadata chain (Hetzner, EC2, GCE, Azure, DigitalOcean). Read only by the rebind gate. Nothing in terraform sets it.ce ent/schema/node.go:68-76, subsystems/impl/node_self_register.go:548-557,765-781
volume locationstamped on the volume row at bind through the optional VolumeLocationReporterce daemon/volume_binder.go:234-247, modules/storage/volumeapi/types.go
move strategyephemeral / rebind / backup+restore, the minimum over every mount. The fast paths relocate only on Swarm; a compose or podman relocation always copies (MOVE-5). The entry point always takes the archive (DNS-2).ce modules/clone/move_strategy.go
rebind gatea block volume rebinds when its volume.location equals the target’s node.location; a proven shared-filesystem mount and a same-node mount also rebind; everything else copiesce daemon/clone_move_adapters.go:268-350,620-660
volume provider choiceexactly one volume-capable connection auto-selects; two or more is X400327VOLAMB, a warning and a host-bind, “no node-location tie-break is wired yet”. Only storage.volumes.provider settles it: the ambiguity branch returns before --storage is read.ce daemon/volume_binder.go:684-748, modules/storage/storageapi/errors.go:596-611
placement enginethe EE scorer ranks nodes for a compose first install (spread, memory headroom) and the install dispatcher runs it there. Swarm gets no score (“SHC owns intent, swarm owns placement”). The scorer “cannot refuse”.modules/placement/advisor.go; ce modules/app/placement_seam.go, modules/app/install_dispatch.go, daemon/services_install_placement.go:49-58, modules/stack/placement_seam.go
--nodeSwarm: a node.hostname== constraint. Compose and podman: a foreign node is refused (X400956NDLOCL).ce modules/app/swarm_placement.go, modules/app/runtime_guard.go:150-200
explainthe scorer’s Decision keeps every candidate’s terms “so an operator can be told WHY”, but ChooseNode returns only the node id, so nothing shows itmodules/placement/placement.go:211-224, modules/placement/advisor.go
location in scoring”node.location exists on the row and the scorer ignores it”docs/proposals/2026-08-11-placement-policy.md:802-805
drainfilters candidates per deployment through the constraint matcher and refuses by name when nothing satisfies a pin; reads constraints from docker service inspect and labels from docker node inspectce modules/cluster/placement.go; modules/cluster/drain.go:1188-1210, daemon/wiring/cluster_drain_constraints.go:73-110
control planerqlite. The voter target defaults to three and is settable (shc cluster set-voter-target); later nodes join as non-voters; suffrage is fixed at join by a voter_slot claim. Swarm managers target a constant three.ce modules/cluster/voters.go, voter_grant.go, types.go:666, swarm_managers.go:81-97
NebulaEE, off by default, 10.42.0.0/16, UDP 4242, every node a lighthouse, CA key on the enabling node’s disk, lighthouse pins in replicated KV, mesh_address persisted on the node row. Nothing SHC dials rides it.modules/nebula/, modules/cluster/mesh_orchestrator_impl.go, daemon/wiring/services_extra_formation.go:66; ce ent/schema/node.go:62; docs/guide/operations/cluster-setup.md:45-48
fabriccluster choice swarm / shc / none, default the Swarm overlay. The shc fabric (WireGuard shc0, IPAM, renderer, nftables compiler) is built; ce runs podman stacks and admits them on --fabric shc. Peers dial internal_address. A Traefik host on the shc fabric is not built.docs/architecture/cluster/network-fabric.md:150-180,872-882; ce core/runtimecmd/runtimecmd.go, modules/fabric/types.go:98-106, subsystems/impl/fabric_reconcile.go:463-467
ingressone system Traefik, a compose project with 80/443 published on one node; every managed record points there or at one edge VIPdocs/architecture/cluster/traffic-across-runtimes.md:294-297,790-800; ce modules/clone/move_ingress.go, modules/app/dns_target.go:299-379
edgesan L4 proxy plane with pfSense HAProxy, Hetzner LB and eight other cloud LB backends; one edge serves the system Traefik; a second web connection is X200981DNSAMBce modules/proxy/proxyapi/types.go, modules/proxy/providers/, modules/app/dns_target.go:328-354
DNSa record in every connection zone that covers the host, each with its plane’s address; private zones minted on demand when exactly one private connection is eligible; private providers are pfSense Unbound, BIND over RFC 2136 and PowerDNS-local; the overlay plane is defined and empty; default TTL 300 sce modules/dns/dual.go, purpose.go, autozone.go, dnsapi/types.go:281-310, modules/app/dns_private.go:105-128, modules/app/dns_target.go:711-749
resolverSHC runs none. Where an estate has a split-DNS resolver today it was installed by hand, outside the product.ce addons/ holds keycloak, openobserve, postgres and traefik only
SSOone Keycloak per cluster in the master tenant, a realm per tenant; the forward-auth sidecar lives in the Traefik project, asks the local daemon, caches an allow for 30 s and fails closedce daemon/services_seams.go:836-841,913-915, addons/traefik/compose.yaml:967-974; forward-auth/cmd/forward-auth/authz_cache.go
backupsbackup.repo resolves per scope through the tenant/environment/stack cascade, “a single global value” in practice; capture runs on the owning node; schedules are keyed by scope, not node; a nightly off-provider mirror existsdocs/guide/operations/enable-backups.md:609-633; ce modules/backup/config_reload.go:112-113, ent/schema/backup_schedule.go
provider LBsa Hetzner LB cannot attach across network zones (X400974PRXLBZ); the Hetzner module’s network is single-zone, “a multi-zone estate is not expressible here at all”ce modules/proxy/errors.go:106-110; terraform/modules/shc/hetzner/variables.tf:333-338

Much of this is EE code in this repo: Nebula, drain, the placement scorer, volume rebind (X403324VOLEEO in CE) and volume failover.

people and devices ── NetBird (+ pfSense VPN) ──┐ routes 10.42.0.0/16
│
site A (home) site B (far)│
┌───────────────────────────────┐ ┌─────────────┴──────────────┐
│ edge A (pfSense HAProxy / LB) │ │ edge B (LB) │
│ Traefik + resolver per node │ │ Traefik + resolver per node │
│ n1 n2 n3 rqlite voters │ Nebula │ n4 n5 non-voters │
│ compose / podman executors │<=======>│ compose / podman executors │
│ keycloak postgres apps │ UDP4242 │ postgres apps │
└───────────────────────────────┘ └─────────────────────────────┘
control-plane fence = site A fabric (shc0) rides Nebula
  1. Executors. Every node runs Docker Compose or Podman. Docker Swarm is retired. Nodes schedule nothing themselves.
  2. One placement engine. SHC places every deployment, at install, move, drain and failover, inside its fences, and explains the choice. Its scheduling mechanics are SHC-1058.
  3. Node addressing. Nebula, on in every multi-node cluster from the first join, is the single plane nodes address each other on. Inside a site Nebula keeps the path on the LAN.
  4. Containers. The shc WireGuard fabric, riding Nebula.
  5. People. NetBird and the pfSense VPN, never in the cluster boot path.
  6. Control plane. rqlite voters live in the control-plane fence.
  7. Per site, an edge. It fronts every healthy Traefik in its location.
  8. Per node, a Traefik and a private resolver. Both read what shcd materialises from the replicated control plane onto the node (SHC-1058’s cluster config directory), so they need no sync with each other.

A fence is a named set of nodes. A node can belong to any number of fences. A rule is attached to an object and says either “stay inside that set” or “share a set of this kind with that partner” (3.3). A deployment’s eligible nodes are the intersection of every rule that applies to it:

eligible(D) = ⋂ fences named by absolute rules on D's scopes, estate to deployment
⋂ reach fence of every volume and bound resource of D
⋂ fences compiled from the relative rules of D's integrations
⋂ nodes that are healthy/degraded and active (today's candidate filter)

An empty intersection is a refusal that names the fence that emptied it.


Every fence is a node set with a kind (a family) and a name. Its id is <kind>:<name>.

originkindsmembership comes fromwho editsused by
built-in, derivednodeeach node, a singletonnobodyany
built-in, derivedprovidernode.provider (new, 3.2)nobodyany
built-in, derivedlocationnode.location, qualified by provider: location:hetzner/hel1nobodyany
resourcevolume, storage, lb, ip, endpointthe resource’s own reachnobodyresource rules (HARD)
systemthe one control-plane fencedefaults to the genesis node’s location fenceoperatorvoter placement
customany kind the operator names: site, metro, continent, compliance, customer, tierthe operatoroperatoroperator rules

Built-ins are derived from facts SHC knows and cannot be removed. location is the provider’s own attach scope (a Hetzner location, an EC2 availability zone), which is the unit block volumes care about, so it is kept as the provider reports it rather than renamed “region”. Qualifying it by provider answers X400327VOLAMB’s objection that clouds use heterogeneous location vocabularies: location:hetzner/hel1 and location:vultr/hel1 cannot collide.

A node whose location is unknown (a Proxmox VM with no metadata and no declaration) belongs to location:unknown. A single-site estate that never declares locations then has one location fence and colocation is trivially satisfied. On a multi-site estate, shc fence ls and explain flag unlocated nodes, because two of them in different places would look colocated.

Resource fences are the only HARD fences:

  • a block volume with a location: the nodes of that location (the rebind gate computes this today, ce daemon/clone_move_adapters.go:620-660);
  • a local volume or host bind: its node (placement ruling P2, docs/decisions/2026-08-14-placement-rulings.md:10);
  • a shared filesystem: the nodes with a proven store_node link (X400340LNKEVD refuses a hand-made link);
  • a provider LB or floating IP: the nodes it can attach to (X400974PRXLBZ for Hetzner network zones);
  • a public endpoint: the nodes holding a public address. A NAT’d node cannot be reached from the internet, and the DNS ladder already refuses a private address for internet exposure (ce modules/app/dns_target.go).

This is the “storage domain” of docs/proposals/2026-08-11-swarm-native-rebalance.md §3.3 (“Local → one node. Cloud volume → the nodes in its AZ. Shared → the whole estate”), generalised to every resource and made a first-class object.

Custom kinds are free-form and may overlap: a node can sit in metro:us-west and metro:pacific at once. There is no hierarchy object. Nesting emerges from membership: fence A is effectively inside fence B when every node of A is in B, so site:ogden inside metro:us-west inside continent:na needs no declaration beyond membership.

Runtime and architecture requirements stay Phase-1 refusals (rulings P6 and P11). They could become derived fences later; nothing here needs that.

facttodaydecided
node.locationSHC_NODE_LOCATION, else cloud metadata (ce subsystems/impl/node_self_register.go:765-781)also declared from terraform (shc_node.location, modules/core nodes[].location) and shc node join --location
node.providernot stored. The Hetzner probe fills CloudMetadata.Cloud, which is not persisted (node_self_register.go:537-546)a new column: SHC_NODE_PROVIDER, else the probe that answered, else declared by terraform or --provider
custom fence membershipn/ashc fence add-node, the shc_fence terraform resource, or a join-time --fence list

Declared beats detected, the order both existing facts already use.

Fences say where nodes are. Rules say where deployments may run. Every rule is attached to a concrete object and travels with it; there are no estate-wide placement “policies” floating free of any object, because those act at a distance and nobody can tell which one moved an app. (This rationale is the only place the word appears.)

A rule is absolute, “stay in fence F”, or relative, “share a fence of kind K with X”.

Rules come from exactly four sources:

sourcethe objectset byform
resourcea volume, floating IP, LB or public endpointits own reach, derivedabsolute
integrationa plug-to-socket bindingthe socket’s default, adjusted by the plug (3.5)relative
deploymentone deployment--rule at install, bind or move; a rules: entry in its deployfile; rules on shc_deploymentabsolute, or a relative override for one of its integrations
scopethe tenant, and the estate, environment and stack on the same ladderthe operator, on that objectabsolute

The owner first named the fourth source “tenant”, then refined it the same day: rules attach along the scope ladder SHC already uses for config, vault paths and backup schedules. That ladder is core/scope (global → tenant → environment → stack → deployment, ce core/scope/scope.go; one ladder by ADR-0003). Reusing it brings the stack rung along with the ones the owner named, and integrations and resources hang below the deployment.

Rules narrow; they never override. Config walks scope.Cascade and takes the first hit, so the most specific value wins. Placement rules walk the same cascade and apply every hit: the eligible set is the intersection of all of them. A lower scope can narrow and can never escape a higher scope’s fence, and inheritance runs only along that ownership ladder.

Loosening is explicit, recorded and authorised. A loosening names the rule it relaxes, carries a reason, and is recorded with its author. Only someone with authority at the scope that set the rule may loosen it: a tenant admin cannot relax an estate rule. Resource rules cannot be loosened by anyone. Today config writes are platform-admin only at every tier (ce modules/config/router.go:35-47), so rules need a finer check than config has. Decision 27 maps rungs to roles: platform:manage at every rung, a tenant’s tenant:manage on that tenant and below.

The sources fall into three classes, by how far their rules bend:

classsourcesoverridableexamples
HARDresourceneverblock volume location; local disk = its node; shared FS = linked nodes; public endpoint = nodes with a public address; provider LB / floating IP attach scope
DEFAULTintegrationyes, explicitly and recorded, because it is performance and not correctnesspostgres, mysql, redis/valkey, mongodb, clickhouse sockets: consumers share location. oidc, saml, smtp, imap, webhook, s3-download, llm sockets: none
OPERATORdeployment, scopeby whoever has authority at that runganything

The DEFAULT class is product behaviour: catalog guidance every estate gets. Which sites exist and which fences and rules an estate uses is estate configuration, never a product default (the owner’s 2026-08-26 ruling that an estate’s operating choices never become product defaults).

The two ends of an integration must share at least one fence of kind K. The socket sets the default, the plug adjusts it, and the operator can override it for one deployment with a recorded --rule colocate=<plug>:<kind>. Relative rules are judged on each end’s eligible set, not its current node, so no later placement decision can break one silently. Resolution, replicas, symmetry and transitivity are in section 5.

The socket declares the default, next to schema and data, naming the fence KIND a consumer must share:

apps/standalone/mysql/mysql.socket.yaml
placement:
colocate: location # node | location | none

Catalog sockets may name only built-in kinds (node, location) or none. Custom kinds differ per estate, and a catalog default naming site would mean nothing where no site fence exists. Custom kinds enter through plug overrides in an estate’s own packages and operator rules at the tenant, environment, stack and deployment scopes.

The plug may adjust it, and only for the services it lists:

apps/enterprise/defectdojo/postgres.plug.yaml
services: # existing field: the consumer services that dial the socket
- defectdojo-uwsgi
- defectdojo-celery-worker
placement:
colocate: node # tighten
# or, to loosen, a reason is mandatory:
# placement:
# colocate: none
# reason: only the one-shot initializer reads it
  • The rule binds only the services the plug lists in services:. The field already exists and already limits which consumer services join the provider network (ce modules/app/integration_render.go:244-252).
  • A plug used only by a provisioning or one-shot job has no runtime colocation: when every listed service is a one-shot, the effective rule is none without the plug saying so.
  • A plug may tighten freely and may loosen only with reason:.
  • The effective rule is the strictest of socket and plug unless the plug loosened explicitly. Among catalog values, node > location > none.
  • The operator overrides at install or bind, either direction, any kind including custom ones: --rule colocate=postgres:metro. The override is recorded on the integration edge and shown by explain.

External sockets. apps/standalone/external-postgres and its siblings front RDS, Cloud SQL or Neon. The provider is not a node, so the socket declares where it lives as a fence id from a var, or none:

apps/standalone/external-postgres/postgres.socket.yaml
placement:
colocate: location
location: "{{ vars.location }}" # e.g. site:aws-us-east-1, or empty

For an external provider the rule is a scored preference, never a refusal: consumers prefer nodes inside that fence. An absolute rule makes it hard.

Catalog lint. The catalog build refuses:

  1. a socket for a data-store interface (postgres, mysql, redis, valkey, mongodb, clickhouse) with no placement.colocate;
  2. a catalog socket or plug naming a kind other than node, location or none;
  3. a plug that loosens without reason:;
  4. a plug on a colocated interface with no services: list;
  5. a services: entry that is not a service in the package’s compose;
  6. an external-* socket without location or with colocate other than none or location.

No socket or plug file carries a locality field today, so the lint lands with the catalog sweep that satisfies it.

Today the binder resolves one provider before it knows where the deployment lands, and two volume-capable connections decline to host-bind with X400327VOLAMB.

Decided:

  1. Each volume-capable connection reports its reach before provisioning: a location-scoped provider its configured provisioning location, a shared filesystem its linked nodes. VolumeLocationReporter allows this (“may ignore handle and return their single configured provisioning location”, ce modules/storage/volumeapi/types.go), and eight of its nine implementations already answer without a handle. Cinder looks the volume up first (ce modules/storage/providers/cinder/provider.go:260-264) and needs a no-handle arm.
  2. The engine picks the node from eligible(D) ∩ ⋃ reach(connection) when the package declares x-shc-volumes.
  3. The binder keeps only the connections whose reach contains that node. One is selected, zero keeps the host-bind path, and two or more is still X400327VOLAMB, now a true tie that storage.volumes.provider settles.

Two Hetzner locations are two connections, each with its own location.

3.7 One placement engine on Compose and Podman

Section titled “3.7 One placement engine on Compose and Podman”

Today’s fabric matrix (network-fabric.md:150-180) has swarm (Compose + Swarm), shc (Compose + Podman) and none (one node). The decided line keeps only the shc fabric: every node runs Compose or Podman, and SHC’s one placement engine places every deployment. Neither runtime has cross-node placement of its own, so nothing competes with the engine.

What the engine does, and what it inherits:

  • Install. It filters candidates to the eligible set, the scorer ranks inside it (modules/placement/advisor.go), and the dispatcher runs the deploy there (ce modules/app/install_dispatch.go). The scorer’s contract stays “it returns a node or nothing; it cannot refuse” (ce modules/stack/placement_seam.go); refusing an empty eligible set is a Phase-1 admission gate before the QUEUED row.
  • --node and --rule. X400956NDLOCL was written because nothing could run a compose deploy on another node. The dispatcher can, so both flags become placement inputs and the refusal goes.
  • Move. MOVE-5 exists for the same reason: only Swarm could relocate by re-rendering. With the dispatcher, a compose or podman relocation is stop on source, rebind if the volume allows, deploy on target, so the ephemeral and rebind fast paths apply to both runtimes.
  • Record. recorded_node_id stays where the bytes are, pinned_node_id stays the operator’s explicit --node, and the rules attached to the deployment and its scopes are the recorded intent every later move and drain reads. Writing a scored node into the pin slot is the mistake modules/stack/placement_seam.go warns about.
  • Explain. Every decision keeps the scorer’s full Decision and the fence provenance (3.17).
  • Scheduling mechanics (rescheduling on failure, replicas and spread, taints and tolerations) are SHC-1058. This design supplies its inputs: the eligible set per service and the move group.

Retiring Swarm removes the Swarm runtime and --fabric swarm, the Swarm placement renderers (ApplyDeployPlacement, ApplyVolumePlacement, ApplyVoterExcludes in ce modules/app/swarm_placement.go), the unread storage.<provider> label helper (ce modules/storage/swarm_label.go), converge-swarm, and drain’s docker service inspect constraint reads. No Swarm shim survives (6.3).

The Swarm keys in package compose files stay, with their meaning (ruled on SHC-1058). Any key that had an effect under Swarm keeps it under SHC or is refused at render, never silently dropped. Today the fabric doc accepts the opposite, “compose already ignores swarm-only deploy: keys” (network-fabric.md:262-268). Two of those keys are placement, so they land here:

  • deploy.placement.constraints translate into rules on the deployment: node.hostname==X to fence=node:X, node.labels.K==V to a custom fence K:V. A constraint with no fence form is refused at render. No catalog package declares one today (a grep of apps/, addons-overlay/ and ce addons/ finds none), so this is for estate and third-party packages.
  • deploy.replicas and mode: global become placement inside the eligible set: N copies spread by SHC-1058’s rules, or one per eligible node.

The rest (configs, secrets, resources, update and rollback, restart) is SHC-1058’s.

Move. For shc move D --node N or --rule fence=F:

conditionoutcome
N outside a fence named by one of D’s absolute rulesrefused by name, naming the rule and the scope that set it
N breaks a colocation rule with a partnerrefused by name, offering shc move --group D --rule fence=<fence>
N inside every volume’s reachephemeral or rebind, on either runtime (3.7)
N outside a volume’s reachbackup+restore, as the rebind gate already decides (ce daemon/clone_move_adapters.go:323-350); the volume is re-provisioned from the connection whose reach covers N (3.6)

Move group. The connected component of colocation edges (5.4). shc move --group D --rule fence=F checks every member’s rules against F, then moves members through the single-deployment Move in dependency order: stop consumers, move providers, move consumers. Nothing is re-rendered for addressing: cross-node names resolve live through each node’s resolver (3.12), which replaces the fabric’s render-time extra_hosts and the re-render ripple it accepted (network-fabric.md:326-331,854-862). It keeps a write-ahead journal like the drain’s (modules/cluster/drain_journal.go), so a failure resumes rather than guesses. A failed member stops the group, leaves the rest at the source, and prints that the colocation is violated with the command that finishes or reverses it. SHC never moves a group on its own initiative.

Drain. The EE drain already refuses by name when nothing satisfies a pin (modules/cluster/drain.go:1188-1210, modules/cluster/drain_pin_gate.go). It reads each deployment’s eligible set from the fence store instead of docker service inspect, refuses a deployment with no in-fence target by name, and never moves voter duty out of the control-plane fence.

Failover. The EE volume failover ticker (off by default, subsystems/volume_failover.go) already stays inside the volume’s location. It also stays inside the deployment’s eligible set; no target means the deployment stays down and the stall diagnosis names the fence. What triggers a failover and how aggressively is SHC-1058, which reuses the dead-source move rather than building a second one. shc move --presumed-dead already relocates off a dead node from replicated state: it needs the node unreachable in the registry and a fresh TCP probe failing, rebinds surviving block volumes behind a generation bump, and refuses local-only mounts unless --discard-local (ce modules/clone/move_presumed_dead.go). The probe cannot tell dead from partitioned; the self-fence covers that (5.8).

Today the voter set is the first nodes to join, up to the voter target, with no location awareness. rqlite fixes suffrage in the join request and “exposes no online role change” (ce modules/cluster/voters.go); the only change is leave-and-rejoin (PromoteBeforeDeparture, ce voter_rebalance.go). The voter slot is claimed at enrol (ce modules/cluster/voter_grant.go). With Swarm retired there are no managers to converge.

Decided:

  • A built-in control-plane fence. Its default membership is the genesis node’s location fence. The operator can change membership; the fence cannot be deleted.
  • A node outside the fence always joins as a non-voter, whatever the voter count. The check runs at enrol, in the voter-slot claim, because suffrage cannot change later without a rejoin.
  • Drain, failover and PromoteBeforeDeparture never pick a candidate outside it, and leader-only work therefore always runs inside it.
  • shc cluster status shows voters against the fence, warns when the fence holds fewer than three healthy nodes, and notes when it has no spare (fewer than four).
  • Refined by Clay on 2026-09-30: the voter count is automatic and odd, the largest odd number up to the fence’s member nodes, capped at five, with automatic promotion and demotion as nodes join or leave (new-line simplification §4.9).

Losing nodes is in 5.9.

What stops the mixed shape today. Measured in code:

  1. Nothing SHC dials rides Nebula. Raft and the rqlite API bind the --nic interface (“Raft does not ride the Nebula mesh”, docs/guide/operations/cluster-setup.md:45-48); peers dial InternalAddrFor, “the only dialable address column node rows carry” (ce modules/cluster/peer_seam.go:110-114); fabric peers dial internal_address (ce subsystems/impl/fabric_reconcile.go:463-467).
  2. Nebula dials internal_address too, so a NAT’d LAN node publishes an unroutable endpoint: “the overlay cannot bootstrap the nodes it exists to bridge” (docs/proposals/2026-08-02-node-addressing-model.md:449-462).
  3. The Nebula CA key lives only on the enabling node’s disk. GetCA reads {state}/nebula/ca.key (modules/nebula/service.go:376-395), “the leader is the only place with the nebula CA key” (modules/cluster/mesh_orchestrator_impl.go:40-41), and EnsureCAExists mints a new CA when none is on disk. The cluster mTLS CA, by contrast, is escrowed in the vault (docs/guide/operations/node-rotation.md:316-319).
  4. Every node is a lighthouse advertising its internal address. A lighthouse pin exists in replicated KV (daemon/wiring/services_extra_formation.go:66) with no terraform surface.
  5. Proxmox nodes are not staged with nebula-cert, the mixed example’s own KNOWN GAP (terraform/examples/mixed-proxmox-hetzner/main.tf:27-37).
  6. The shipped config is WAN-hostile: MTU 1300, punchy on, no preferred_ranges, no relays (modules/nebula/templates.go). Until SHC-1049 it also shipped tx_queue: 500 and default socket buffers: one TCP stream over a 150 ms link reached 20 Mbit/s with those and 147 Mbit/s with 16 MiB socket buffers and tx_queue: 5000, which now ship (network layering decision). The same cell cut everything but UDP 4242: the far node lost quorum while its mesh pings answered, and healed about 50 s after the path reopened (Swarm about 2.5 min), which is item 1 seen from a rig.
  7. Latent: NIC auto-detect excludes an overlay device named nebula1 (ce core/nic/nic.go:357-361), but the device SHC creates is nebula0 (modules/nebula/templates.go:131).

Decided shape.

  • Always on. Every multi-node cluster runs Nebula from the first join; a one-node estate has no peer to address. It is the single node-addressing plane: raft (-raft-adv-addr, docs/proposals/bind-advertise-split.md), the rqlite API, the internal API and fabric peers all dial mesh addresses. There is no LAN-addressed mode beside it.
  • The addressing model gains a fifth role. Its rule “every SHC-owned overlay dials the internal address plus its own port” cannot hold when the internal address is the overlay. underlay is what Nebula dials (the LAN address inside a site, the public or declared edge address across sites); internal is the mesh address.
  • Bootstrap over the underlay, run over the overlay. A joiner enrols against a lighthouse’s underlay address (the internal API is mTLS), receives its Nebula cert, the CA and the lighthouse list in the enrol response, starts Nebula, and addresses everything by mesh from then on. Joins already ride “the LAN token over any routable path” (modules/node/commands/join.go:39-47).
  • The Nebula CA lives in the vault, like the cluster CA, so any voter can sign.
  • Lighthouses and relays are the nodes with a public address, at least one per site that has one and at least two in total, pinned from terraform. LAN-only nodes are members that punch out through them; relays cover symmetric NAT.
  • preferred_ranges is each node’s site LAN, so same-site traffic stays on the LAN path.
  • WAN tuning of buffers and tx_queue from SHC-1049’s final numbers.

The shc fabric carries container traffic: WireGuard shc0 per node, a /24 per node, pinned service addresses, and today cross-node names baked into extra_hosts at render time (network-fabric.md:1-40,970-989). Its peers dial mesh addresses, so it rides Nebula; its MTU drops from 1420 to 1220 (Nebula’s 1300 less WireGuard’s 80).

Ruled on SHC-1058: a stack renders as one compose project per node it touches, holding only the services placed there, and cross-node names resolve live through each node’s resolver (3.12) instead of extra_hosts. A move no longer re-renders the stacks connected to the moved one.

Two things must land before Swarm can go, because today the Swarm overlay does them:

  • Traefik on the fabric. “A Traefik host on the shc fabric is slice 2 work” (network-fabric.md:878), and the renderer bakes no ingress host entry, so on --fabric shc “the firewall permits traffic to a target the proxy has no way to name” (traffic-across-runtimes.md:352-380). With Traefik’s routes fed from the control plane (3.13) its backends are fabric addresses, so nothing has to be baked.
  • Fabric slices 2 and 3, tracked as SHC-1055 (nftables filtering and default-drop) and SHC-1056 (Podman adapter, depends on SHC-1055). The fabric doc disagrees with itself about how much of both landed (stale docs, 6.6), and ce already carries an nftables compiler (modules/fabric/policy/) and runs podman stacks, so both tickets start by stating exactly what is missing.

The fabric doc’s “Retire the Nebula module” and “WAN membership over the fabric” follow-ons (network-fabric.md:1083-1098) are withdrawn: Nebula addresses nodes, the fabric carries containers.

Which provider owns which zone. A domain’s public zone lives at one public DNS connection (Hetzner, Cloudflare and so on). A domain resolves privately only if it is on a private connection’s zone list; SHC then mints the private zone (autozone.go) and publishes a record in every zone that covers the host, each with its own plane’s address (dual.go). Records are provider-created per connection, never by hand.

Decided answers.

  • The public answer is the edge of the location the engine placed the app in (3.13): that location’s LB or pfSense VIP. A location with no edge answers with one healthy node that holds a public address, re-pointed by the engine when it fails (decision 28).
  • The private answer for a routed hostname is the mesh addresses of the healthy Traefiks in that location. It is the same answer everywhere, and it is reachable from every node over Nebula, from every container through its host, and from every NetBird device through routing peers that advertise 10.42.0.0/16.
  • A service name between containers answers with the fabric addresses of the healthy replicas (SHC-1058).

When an app moves between locations. The engine re-runs the DNS ladder for the moved deployment and the public record re-points at the new location’s edge. Clients cache the old answer for up to the record TTL, 300 s by default and settable per scope with dns.ttl (ce modules/app/dns_target.go:711-749). That window costs nothing: every Traefik carries every route (3.13) and reaches every backend over the fabric, so the old edge keeps serving across the WAN until caches expire. Certificates are already on every Traefik (3.13), so the new edge needs no fresh issuance. Lowering dns.ttl before a planned move shortens the hairpin. The private answers change as soon as the new placement is materialised on each node.

A resolver on every node. Each node runs a private resolver that answers from what shcd materialises out of the replicated control plane onto that node, the cluster config directory of SHC-1058. There is no zone transfer and no sync between resolvers. The node’s host resolver asks it, and containers reach it through their runtime’s embedded resolver, which forwards to the host’s. During a partition it keeps answering from the last materialised state, so a cut-off site still resolves its own names. NetBird devices reach the node resolvers’ mesh addresses through a NetBird nameserver group. The resolver is a container in the builtin Traefik project beside forward-auth (decision 26).

The private DNS connections stay. They still decide which domains are private, and pfSense Unbound, BIND and PowerDNS-local keep the LAN-plane records SHC publishes into them for clients that ask them directly, such as a house LAN behind a pfSense edge.

state
works todaydual publication writes every matching zone with its plane’s address (dual.go); private zones are minted on demand (autozone.go); LAN-plane records carry dns.lan_address, else the public answer, for a site’s LAN clients (dns_private.go:105-128); the public record is always kept, so an off-VPN client gets a clear refusal rather than NXDOMAIN (purpose.go); dns.ttl is a scoped knob; moving the entry point re-points every managed record (ce modules/clone/move_ingress.go)
wronga private name has one answer, reachable from one site’s LAN only; nothing serves the overlay plane, and dns_private.go’s note that “the node table carries no overlay address” is stale for Nebula, which persists node.mesh_address; two private connections make autozone refuse (X200923DNSAZA, autozone.go:27-31), which an estate with a pfSense edge in each site trips
missinga public target per location (today one Traefik and one edge); a resolver on every node answering mesh and fabric addresses from replicated state; node and container resolver configuration; NetBird nameserver groups, which nothing manages (SHC-277 is the future NetBird DNS lane, ce modules/connections/capabilities.go:297-298)

Decided: one edge per location fence, a pfSense HAProxy or a provider LB, in front of every healthy Traefik in that location. The edge reaches them over the location’s LAN, which is why it is per location.

Ingress: a Traefik on every node (ruled on SHC-1058). There is no Traefik-to-Traefik sync. Hostnames, healthy backends and their fabric addresses live in the replicated control plane; shcd materialises each node’s share into the cluster config directory, and the local Traefik reads it through its file provider, which the builtin already runs (--providers.file.directory=/etc/traefik/dynamic; the daemon sends a SIGHUP after writing, because the provider’s watch skips subdirectories, ce addons/traefik/compose.yaml:638-647, modules/app/traefik_reload.go). The Docker-label provider (:624-628) goes. Every Traefik carries every route and reaches every backend over the fabric, preferring backends in its own location, so DNS alone decides which location a client enters and a re-point never waits for a re-render. Readiness gating and draining cover the seconds a node’s copy lags. The forward-auth sidecar comes with each Traefik, since it is part of the Traefik project. The fabric’s nftables compiler already models a per-node Ingress class (ce modules/fabric/policy/model.go).

Certificates are issued once and distributed through the control plane; no Traefik runs ACME. Today Traefik on the one ingress node does ACME, and the cert-delivery broker reads its store to hand certificates to apps that terminate TLS themselves (ce modules/certbroker/doc.go, modules/certbroker/acme_source.go). The broker’s source seam already names the replacement: “a daemon-side ACME client is a second implementation behind this same seam” (ce modules/certbroker/broker.go:14-15). With one, the issued material flows to every Traefik and to the broker’s app deliveries alike. The entry-point move (ce modules/clone/move_ingress.go) has nothing left to move and retires.

Apps with replicas in several locations. Three shapes:

  • One global entry point. The app’s public name points at one location’s edge; its Traefiks proxy to replicas in every location over the fabric. Nothing new is needed beyond the per-location edge. Losing that location loses the entry point.
  • Health-checked DNS. SHC probes each location’s edge and withdraws a dead one’s records, so a client fails over within a TTL. SHC’s DNS module has no health checks today.
  • GeoDNS. Answers by client location. Provider-specific and usually a paid feature; SHC has no support for it.

The v1 answer is one global entry point, because colocation keeps stateful apps in one location anyway; health-checked DNS is the follow-up, and GeoDNS is not planned.

Hard fences. A provider LB attaches only within its own scope: a Hetzner LB cannot cross network zones (X400974PRXLBZ), and the Hetzner module cannot express a multi-zone network (terraform/modules/shc/hetzner/variables.tf:333-338). Each LB is a resource fence of its location. SHC manages no movable floating IP today (the OpenStack module’s per-node floating IPs are NAT addresses, terraform/modules/shc/openstack/main.tf:207-221); when it does, each is a resource fence with its provider’s attach scope.

Edge failover. A provider LB is highly available inside its location, and that is the provider’s job. A pfSense edge is one box: SHC has no CARP or HA pair support (nothing in ce mentions CARP, VRRP or keepalived). When an edge dies but its location lives, the other location’s edge can take the public names and reach the apps over the fabric; when the location dies, backups are the recovery path (3.16). Until health-checked DNS exists, that re-point is a runbook step.

state
works todaythe L4 proxy plane: pfSense HAProxy, Hetzner LB, AWS NLB, GCP TCP LB, Azure, DigitalOcean, Linode, Vultr, Octavia and CloudStack backends (ce modules/proxy/providers/), shared-VIP and per-app cardinality, full-pool reconcile, the VIP handed to DNS (proxyapi/types.go); PROXY protocol with the edge’s own addresses kept out of LAN allowlists (proxyapi/edge.go); Hetzner LB targets resolved to servers and attached to the private network (providers/hcloudlb/provider.go)
wronga second web connection on the system Traefik is X200981DNSAMB, so there is one edge per cluster (ce modules/app/dns_target.go:328-354); a compose Traefik publishes on one node, so the edge’s pool narrows to that node (Forward.Pool, proxyapi/types.go)
missingan edge per location fence, with DNS choosing by the app’s location; Traefik on every node with routes fed from the control plane; a daemon-side ACME source for the cert broker; health-checked DNS; floating IPs; pfSense edge HA

The rule of thumb is calls per user request times round-trip time. At 150 ms RTT a page that makes 50 sequential database queries waits 7.5 s.

integrationdefaultwhy
postgres, mysql, redis/valkey, mongodb, clickhouselocationper-query round trips on the request path
the app’s own filesHARD, via its volumethe volume’s reach fence
oidc, samlnonea few round trips per login, none per request
smtp, imapnoneasynchronous or per-session
webhooknonefire-and-forget
s3 downloads, llmnonefew, large requests
app-to-app APIs on the page path (webdav, document editors, PDF tools)the socket author’s call, usually locationchatty in practice

Today one Keycloak serves every realm (ce daemon/services_seams.go:913-915). The forward-auth sidecar in each Traefik project asks the local daemon per session, caches an allow for 30 s, fails closed, and serves stale answers only on a 429 (forward-auth/cmd/forward-auth/authz_cache.go).

Decided for v1: one Keycloak in the home site, OIDC colocation none. A far-site login pays a few WAN round trips. During a partition, far-site gated apps fail closed within about 30 s, because the daemon’s answer needs leader-forwarded reads (5.8) and Keycloak is across the cut. A follow-up lets the daemon answer forward-auth from its local replica with bounded staleness, so established sessions survive a partition; new logins still need Keycloak. Keycloak per site means Keycloak cross-site replication and is out of scope.

Decided.

  • Targets are chosen per location, off-site. Each location fence carries its own backup target as an attribute of the fence object (shc fence set location:hetzner/hel1 --backup-repo …). A deployment’s archives go to the target of the location it runs in unless a narrower config scope sets backup.repo, and each location’s target lives outside that location, so losing a location never loses its backups.
  • A second copy on a second provider. The nightly off-provider mirror (enable-backups.md:609-633) points at Backblaze B2 or another provider that is not the first repo’s. The estates have no true second provider today: their object storage is Hetzner only.
  • Restore-as-move is the copy path. A move that leaves a volume’s reach is backup+restore already (SelectMoveStrategy, MOVE-2 and MOVE-6); it carries the re-provisioning of 3.6.
  • Backups are the recovery path when a location is lost. Each lost deployment is restored from its off-site repo into a surviving location with shc move --presumed-dead (ce modules/clone/commands/main.go:298-315) or shc recover, with its fence pointed at the new location. The recovery point is the schedule interval.
  • The schedule follows the app. Schedules are keyed by scope (tenant, stack, environment; ce ent/schema/backup_schedule.go), not by node, so a move changes nothing about when backups run. Where they go follows the new location’s repo; each backup row records its archive’s location (backup.location, ce ent/schema/backup.go), so history in the old repo stays restorable.
state
works todayper-scope backup.repo; capture on the node that owns the volume; the verify sweep; the off-provider mirror mechanism; shc recover and --presumed-dead moves; backup+restore as the cross-reach move; scope-keyed schedules; per-archive location on the backup row
wrongbackup.repo is “a single global value: one provider, one credential, holding every archive the estate has” (enable-backups.md:611-614); there is no site axis, so a far-site capture crosses the WAN to one bucket; the mirror is armed on the leader only; no true second provider on the estates
missingper-location off-site repo resolution; a per-location mirror; a location-loss runbook proven on a rig

shc fence explain <deployment> prints the eligible set and every rule that shaped it, each with the object and scope it came from, the move group, the chosen node with the scorer’s terms (modules/placement/placement.go:211-224), and for a refusal the first fence that emptied the set:

$ shc fence explain acme/nextcloud/prod
deployment acme/nextcloud/prod compose
running on gor-02 scored 812/1000 (spread 640, memory 930)
eligible gor-01 gor-02
resource volume nc-data stay in storage:ceph-ogden gor-01 gor-02 gor-03
integration postgres plug -> acme/postgres/prod
share location (socket default) location:gorillaservers/ogden
tenant acme stay in site:us-west (2026-10-02) gor-01 gor-02 hil-01
deployment acme/nextcloud/prod no rules
move group acme/nextcloud/prod, acme/postgres/prod (location)

Every placement refusal (install, move, drain, fence edits) prints the same lines.

  • CLI: shc fence create <kind>:<name>, ls [--kind], show, add-node, remove-node, rm, explain. rm refuses built-ins and fences a deployment depends on; remove-node refuses when a deployment on that node would leave its eligible set, naming it.
  • Rules at a scope: shc rule add <rule> / ls / rm with the same scope flags shc config set takes (--tenant, --environment, --stack), plus --reason for a loosening.
  • Rules on one deployment: --rule (repeatable) on install, update, bind and move, in two forms, --rule fence=<fence> and --rule colocate=<plug>:<kind|none>; a rules: entry in a deployfile. --rule fence=location:hetzner/hel1 is the explicit form the ticket called --region. shc move --group moves a move group.
  • Membership: --location, --provider and --fence on shc node join (the fences the node joins).
  • Terraform: a shc_fence resource (kind, name, nodes); a shc_rule resource at a scope; rules on shc_deployment; location and provider on shc_node and modules/core nodes[]; lighthouse pins on shc_mesh.
  • API: the same through the daemon; the openapi document regenerates.

S1. Joining a GorillaServers VM to hel1. Terraform declares location = "ogden", provider = "gorillaservers" and fences = ["site:ogden"]. The VM enrols over a hel1 lighthouse’s public address, receives its Nebula identity, and joins as a non-voter because it is outside the control-plane fence (location:hetzner/hel1). shc fence ls shows location:gorillaservers/ogden with one node.

S2. An app in Ogden whose database is in hel1. shc install wiki --rule fence=site:ogden --plug postgres=acme/postgres/prod is refused: the postgres socket colocates location, the provider’s eligible set is location:hetzner/hel1, and no location fence holds both. The refusal offers three ways out: a postgres in Ogden, a group move, or --rule colocate=postgres:none with a reason (recorded, shown by explain).

S3. Moving a database with its consumers. shc move acme/postgres/prod --rule fence=site:ogden is refused, naming its three consumers. shc move --group acme/postgres/prod --rule fence=site:ogden moves all four by backup+restore (every volume leaves its location), consumers stopped first and started last. Their public records re-point to the Ogden edge; hel1’s edge serves the stragglers for one TTL.

S4. Keycloak in hel1, apps everywhere. Allowed. OIDC colocates none, so no edge forms and Keycloak does not glue the estate into one move group.

S5. A fence spanning two sites. The operator creates metro:us-west with the Ogden and Hillsboro nodes and adds the rule fence=metro:us-west to tenant acme. Every acme deployment inherits it. A stateless acme app may run in either location; how its replicas spread is SHC-1058. Its redis binding (location) puts the app and redis in one location inside the metro.

S6. Draining an Ogden node. Each deployment’s candidates are the other Ogden nodes in its eligible set. A deployment with none is refused by name, and the output names the group move that would evacuate it.

S7. Two Hetzner locations, two volume connections. An install lands on a fsn1 node, and the binder keeps the connection whose reach is location:hetzner/fsn1. No X400327VOLAMB.

S8. The WAN link drops. See 5.8.

S9. A control-plane node dies. See 5.9.

S10. The Ogden location is lost. Its deployments are restored from their off-site repos into hel1 with --presumed-dead, their fences re-pointed, and DNS follows the new placement.


For deployment D, per service s:

eligible(D, s) = ⋂ { F : F an absolute fence of D }
⋂ { reach(v) : v a volume or bound resource of s }
⋂ { F_K(g) : g the move group of D, K a colocation kind binding s }
⋂ healthyActive

healthyActive is today’s candidate filter (cluster_state healthy or degraded, availability active; modules/placement/advisor.go). By default the services of one stack share an implicit node colocation, so the stack lands on one node and its per-service sets intersect into one. Only a service that declares replicas or its own placement is split out (5.7).

Every rule applies at once. Resource rules are never loosened. An integration rule can be loosened only by the plug with a reason, or by a recorded deployment-scope --rule colocate. An operator rule can be loosened only by someone with authority at the scope that set it. Everything else only narrows: an operator rule that would widen a resource fence simply intersects down to it, and explain says so.

“C and P share a fence of kind K” holds when some fence F of kind K contains every eligible node of both ends. Eligible here means the base set, before any derived fence is added. The engine compiles the rule to an absolute fence:

  1. Candidates are the K fences that contain eligible(P) and intersect eligible(C).
  2. None: refused by name. If eligible(P) itself spans several K fences, the refusal offers an absolute rule on P or a group move.
  3. One or more: the smallest by node count wins, then by id. The choice is recorded on the move group as a derived fence with its provenance (“from C → P, kind K”) and shown by explain.
  4. The derived fence joins both ends’ eligible sets.

none is the whole-cluster answer. It is trivially satisfied and forms no edge.

Every replica must share a K fence with the partner’s placement. Because the rule compiles to one fence F that is intersected into both ends’ eligible sets, every replica of both ends lands inside F. A three-replica app colocated with a single-writer database runs all three in the database’s location. Spreading them inside it (maximum skew), a stable identity and volume per replica for clustered stateful apps, a fabric address per replica and Traefik’s list of healthy replicas are SHC-1058.

5.4 Symmetry, transitivity, the move group

Section titled “5.4 Symmetry, transitivity, the move group”
  • Symmetric. Moving either end re-checks the rule for both, because the derived fence belongs to the group, not to one end.
  • Transitive. AB and BC (both location) put A, B and C in one location. The engine builds groups with union-find per kind over the binding edges, one derived fence per kind per component.
  • Mixed kinds. AB (location) and BC (metro) give a location for {A, B} and a metro for {B, C}; B sits in both, and the resolver checks that the two choices intersect on B.
  • The move group is the connected component over edges of every kind except none.

Nothing special. “Share a fence of kind K” is evaluated on membership. If every location sits inside some metro, sharing a location implies sharing a metro, because the node sets say so. A node in two metros satisfies a metro rule with a partner in either. A node in no fence of kind K cannot satisfy a K rule, and explain names the missing membership.

All in the replicated store, all in the new line’s single initial migration (6.3):

  • node.provider, nullable text. node.location exists.
  • fence(id, kind, name, origin, description, created_at); origin is derived, resource, system or custom.
  • fence_member(fence_id, node_id, source); built-in and resource rows are written by a reconciler from node and resource facts, custom rows by the operator.
  • placement_rule(id, scope, scope_key, tenant_name, stack_name, environment, form, fence_id, partner, fence_kind, loosens, reason, actor, created_at), keyed on the scope columns and scope_key exactly as backup_schedule is (ce ent/schema/backup_schedule.go), so rules reuse core/scope rather than a parallel ladder. Integration rules live on the edge (below); resource rules are derived and never stored.
  • A location fence’s backup target, an attribute of the fence (3.16).
  • On the persisted integration edge: colocate_override, override_reason, and the derived fence id with its provenance.
  • placement.colocate, placement.location and placement.reason in the socket and plug schemas.

The engine decides a node per service, one node per stack by default (5.1). For each node a stack touches, SHC renders a compose project holding only the services and replicas placed there, under the stack’s own project and network names, and the dispatcher runs it on that node (ruled on SHC-1058). Pinned fabric addresses stay; partners’ names are not baked in, because each node’s resolver answers them live (3.12). Swarm placement keys arrive as rules (3.7). There is no scheduler to render constraints for.

Voters all sit in the home site (5.9). rqlite forwards weak and strong reads to the leader (core/db/rqlitedriver/discovery.go:47-58); SHC’s DSN sets no read level (dialDSN, same file), so reads run at gorqlite’s default, weak (gorqlite conn.go:259 at the pinned version). A far-site daemon therefore loses reads as well as writes.

home sitefar site
control-plane writesworkrefused: no leader reachable
control-plane readsworkfail, forwarded to an unreachable leader
running containersrundeployments on host binds and local disks run; a deployment holding an SHC-managed RWO volume stops itself, all of its containers (below)
placementnormal, but no far node is a candidatefrozen
ingress and edgeserve their hostsserve their hosts from the last materialised routes; route changes frozen
public DNSunchanged (records live at the providers)unchanged
private DNSeach node’s resolver answerseach node’s resolver answers from its last materialised state
SSO-gated appsworkfail closed within about 30 s (3.15)
backupsruncannot record; resume after the heal
volume failoverfinds no candidate for a far deployment, because home nodes are outside its volume’s location fencenothing runs
healnothing to reconcile in raftnon-voters catch up from the log; a stopped writer starts only after the control plane confirms the node still owns it and its lease is re-acquired (SHC-1058)

SHC-1049 measured the heal at about 50 s for SHC, and about 2.5 min for Swarm, after a mesh-only cut (network layering decision).

The self-fence stops far-site writers. Every node runs a self-fence loop, whatever the auto-failover flag says (ce subsystems/impl/volume_self_fence.go:14-22,44-47). Every 5 s it renews a lease on each RWO volume attached to it, and a renew is a quorum write. After two failed ticks (:68-82) it stops every container carrying the owning deployment’s labels, lazily unmounts and flushes (:19-20,576-583; volume_self_fence_node.go:134-177). A far node cut off from the leader cannot commit a renew, so every far-site deployment holding an SHC-managed RWO volume stops within seconds, although no home node could attach its location-bound volume anyway. A healthy renew only clears the loop’s mark (:258-265); the loop restarts nothing it stopped. Host binds and local disks carry no attachment and keep running. Decision 24 closes this gap with end state 7: a volume whose reach holds no node of the control-plane fence is exempt from the self-fence and is taken over only on a STONITH confirm or an operator’s --presumed-dead.

The same loop is what makes the EE automatic failover safe against a node that is only partitioned: that engine waits a rebind dwell of at least 45 s (ce modules/storage/storageapi/errors.go:552-556), and the self-fence finishes well inside it. --presumed-dead has no dwell. It needs the poller’s unreachable stamp, three failed polls on a 2 s tick (ce modules/health/node_poller.go:287-288,396-397), and a failing probe, so when SHC-1058 reuses it for automatic failover it has to wait out the self-fence first (decision 25). On return, SHC-1058 adds the other half: runtime restart policies no longer start SHC-managed containers at boot, and the daemon starts them only after the control plane confirms what the node still owns.

Nothing split-brains: non-voters cannot be elected, so the far site cannot form a second leader. The unsafe verb is shc cluster repair --force (it posts to the force_leader route, modules/cluster/commands/admin_fleet.go:98-123), an operator act against nodes confirmed destroyed. Whether the engine ever fails a far deployment over into the home site on its own is SHC-1058’s call; the fences already forbid it for anything bound to a far volume.

V voters need floor(V/2) + 1 alive. With three in the fence:

voters lostlive votersneededstate
032healthy, tolerates one loss
122writes work, zero fault tolerance (NoFaultTolerance)
212wedged: no writes, and raft cannot remove the dead voters (ce modules/cluster/voters.go, “The quorum-loss deadlock”)

Recovery from one loss: with a healthy non-voter already in the fence, removing the dead voter with --rebalance runs PromoteBeforeDeparture, 3 → 4 → 3, never below target. A fence with only three nodes has no spare, so the removal needs --accept-deficit (a plain removal below target is refused, X409205CLSRMD); quorum 2 of 2 still commits it, the slot frees, and a replacement joining inside the fence takes it as a voter. That window has zero fault tolerance, which is why status notes a fence with fewer than four nodes.

Superseded on the new line (Clay, 2026-09-30): with the automatic odd voter count, removing the dead voter is a plain shc cluster rm, the two survivors become one voter and one non-voter, and --accept-deficit goes (new-line simplification §4.9).

Recovery from two losses is shc cluster repair --force on the survivor, only after the lost nodes are confirmed destroyed.

Losing the whole home site leaves the far site with no voter. Whether repair --force can seat a non-voter as the single voter is not verified; a site-loss runbook proven on a rig is a build step.

Two other layouts were weighed. One voter per site across three sites survives any single site loss and pays WAN latency on every write. A 2 + 1 split across two sites keeps quorum in the home site through a partition, as 3 + 0 does, and gives the far site a voter to recover from if the home site is destroyed; its cost is that raft may elect the far voter, after which every write crosses the WAN and leader-only work runs in the far site. 3 + 0 was chosen until the runbook proves recovery from a non-voter.


fence is the placement primitive. The volume-attachment I/O fencing is renamed lease: the fence generation becomes the lease generation, self-fence becomes lease loss. “Lease” already names nearby things, so the renamed concept is always the attachment lease. volume_attachment already carries owner_ref and leased_until for backup leases on the same row, which fits; deployment_lease, volume_mount_lease and crons_lease are unrelated and keep their names. The STONITH agent keeps its industry name (Fencer becomes a STONITH agent, without the word fence).

The rename lands whole, with no aliases, in the clean line (6.3). Its surface, measured on both mains:

surfacewhat changes
ce Go126 files contain “fence” (case-insensitive), 1,145 occurrences; 78 of them are “defence” and a few more are unrelated senses
shc Goabout 50 files use the volume sense (FenceGeneration, BumpFenceAndAttach, self-fence, fence_pending, Fencer, volume_fence*, assert-node-fenced)
shc docs32 files outside evidence/ and handoff/ name the volume-sense identifiers, not counting this one (64 contain “fence” in any sense)
schemavolume_attachment.fence_generation (ce migrations/20260819001215_auto.sql:1035); the fence_pending value of node.cluster_state (ce ent/schema/node.go:60), which the scorer’s candidate filter names
CLI and APIshc volume assert-node-fenced and /api/method/shc.storage.volume.assert-node-fenced; fence_generation in shc volume ls -o json (ce modules/volume/commands/list.go:36); openapi VolumeAssertNodeFenced{Request,Response} (docs/reference/api/openapi.json:14838-14891)
error codesX409035VOLFNC (“volume … fence refused”, ce modules/storage/storageapi/errors.go:418)
subsystems and filesvolume_self_fence, volume_self_fence_node, volume_fence_override.go; the FENCE-0..3 labels in move_strategy.go

The owner approved every recommendation below and added decisions 16 to 23. Decisions 9 and 10 were refined the same day on SHC-1058.

  1. Adopt the node addressing model (2026-08-02-node-addressing-model.md) as the address vocabulary, with its Q2 answered “declare the address” and one amendment, the fifth role underlay (3.10). Its Q10 (voter removal) is answered by --rebalance; its Q12 (terraform suffrage) by the control-plane fence. Decided: adopted, with the amendment.
  2. What does the cluster plane dial between sites? Mesh addresses, rather than site-to-site routing through gateway nodes or public addresses behind firewalls. Decided: mesh addresses, always (decision 17).
  3. Where does the Nebula CA live? Decided: in the vault; any voter signs.
  4. Do the fabric doc’s “retire Nebula” and “WAN membership over the fabric” follow-ons stand? Decided: withdrawn.
  5. Which kinds are built in? Decided: node, provider, location; an unknown location is location:unknown.
  6. Where do colocation defaults live? Decided: on the socket; the plug adjusts for its listed services; strictest wins unless loosened with a reason; catalog files name only node, location or none; the six lint rules of 3.5.
  7. What do relative rules mean? Decided: judged on eligible sets, compiled to one derived fence per kind per move group, binding every replica; none forms no edge.
  8. Where do voters live? 3 + 0 in the home site, against 2 + 1 or one per site. Decided: the control-plane fence, defaulting to the genesis location, enforced at enrol.
  9. How many ingresses? First decided as one Traefik per location fence. Refined on SHC-1058: an edge per location fence in front of a Traefik on every node; routes and certificates come from the control plane; every Traefik carries every route; DNS follows placement.
  10. Which private resolver? First decided as a PowerDNS-local catalog app fenced to each site, with pfSense Unbound where a site has a pfSense edge. Refined: a resolver on every node answering mesh and fabric addresses from the replicated control plane, running in the builtin Traefik project (decision 26).
  11. SSO? Decided: one Keycloak in the home site for v1; far-site gated apps fail closed during a partition; a local-read forward-auth answer follows.
  12. Backups? Decided: a backup target per location fence, off-site, plus a second copy on a second provider.
  13. Group moves? Decided: explicit (shc move --group) and offered by refusals; never automatic.
  14. X400956NDLOCL? Decided: lifted; --node and --rule go through the dispatcher.
  15. Multi-location apps? Decided: one global entry point in v1, health-checked DNS next, no GeoDNS.
  16. Retire Docker Swarm. SHC’s one placement engine is the only scheduler; nodes run Compose or Podman as executors. Scheduler mechanics are SHC-1058.
  17. Nebula always on as the single node-addressing plane.
  18. The fabric carries container traffic; NetBird is for people only.
  19. No backwards compatibility. A clean line: one squashed initial migration, no in-place upgrade inside the product, no Swarm or LAN-addressing shims (6.3).
  20. Fences and rules, no free-floating placement settings. Every rule is attached to a concrete object and travels with it. There are four sources: resource, integration, deployment, and the scopes above the deployment (tenant, refined to the whole core/scope ladder). Rules narrow downward: all of them apply, a lower scope can never escape a higher scope’s fence, and loosening is explicit, recorded and allowed only with authority at the scope that set the rule. Explain prints every rule with its source (3.3).
  21. Swarm compose keys keep their meaning (SHC-1058). Each is honoured or refused at render, never ignored; placement constraints become rules (3.7).
  22. Per-node compose projections (SHC-1058). A stack renders as one project per node it touches, and names resolve live through each node’s resolver (3.11, 5.7).
  23. Failover reuses the dead-source move, and a returning node starts nothing on its own (SHC-1058). The daemon starts SHC-managed containers only after the control plane confirms ownership and the lease is re-acquired (3.8, 5.8).

Decisions 24 to 28 were the questions the SHC-1058 follow-up left open. They were decided by agent on Clay’s standing delegation, 2026-09-29, reversible. Each took the recommended option; the rejected options are kept so a reversal knows what it is choosing between.

  1. Far-location volumes keep running through a WAN cut. A volume whose reach holds no node of the control-plane fence is exempt from the self-fence and is never taken over on lease expiry. A takeover needs a STONITH confirm (the Hetzner fencer powers the server off and reads it back, ce modules/storage/providers/hcloud/fencer.go) or an operator’s --presumed-dead. There is no automatic failover inside a far site. Rejected: keeping today’s stop, which takes far-site stateful apps down for every cut (5.8), and a lease authority per site, a second consensus group per site that the 3 + 0 layout exists to avoid.
  2. Automatic failover waits for the old node to stop itself. When SHC-1058’s automatic failover reuses --presumed-dead, it takes over a volume only after the self-fence window has passed. --presumed-dead alone has no dwell (5.8); the EE engine’s rebind dwell of at least 45 s is the model.
  3. The per-node resolver is a container in the builtin Traefik project, beside forward-auth, reading the cluster config directory. The builtin set stays at four. Rejected: inside shcd, where every daemon restart or roll drops DNS for every container on the node, and a fifth builtin, which reopens the set ruled closed at four (docs/decisions/2026-08-18-builtin-vs-deploy-placement.md).
  4. Rule authority. platform:manage sets and loosens rules at every rung. A tenant’s tenant:manage sets rules on that tenant and its environments, stacks and deployments, and loosens only rules set at those rungs. Nobody loosens a resource rule. Both roles exist in Keycloak today (ce modules/auth/keycloak_admin.go:2064-2066).
  5. Public DNS for a location with no edge answers with one healthy node that holds a public address, chosen by the engine and re-pointed when it fails, until health-checked DNS lands. Rejected: an A record per public node, where a dead node fails a share of clients until it is withdrawn.

The new line carries zero backwards compatibility. The rqlite and SQLite control-plane migrations are squashed into one initial migration that already holds the fence tables, node.provider and the lease rename. The product cannot upgrade a cluster from today’s line in place, and it carries no shim for Swarm or for LAN-shape addressing.

The migration covers bitspur only for now. The owner ruled on 2026-09-29 that promanager stays on its current line until he decides otherwise, so nothing in this plan touches the promanager estate. bitspur moves with a one-off, throwaway migration script that lives outside the product (not in shcd, and no compat code anywhere in it), runs once, and is deleted once bitspur is across and the owner has settled promanager’s path. It is tracked as SHC-1061. Two ways to write it:

  • A. Convert in place. Stop the control plane, read today’s rqlite state, write it into the squashed schema, lay down the new line’s packages, bring Nebula and the fabric up, and re-place every stack through the new engine. Fast and needs no second set of servers. The risky part is the conversion itself: every table’s semantics has to map, and a half-converted cluster has no product to fall back on.
  • B. Build a fresh cluster and move every app. Stand up the new line on new nodes (or on nodes drained out of the old cluster one at a time), then script backup on the old cluster and restore on the new one, app by app, in move-group order, re-pointing DNS as each lands. Slower and needs capacity, but the old cluster keeps serving until the last app is across.

Either way the script:

  • has a dry run that prints every step, every stack in move-group order and every DNS change, and changes nothing;
  • takes a verified backup of every stack and of the control-plane state before its first write, and refuses to start if any verify fails;
  • has a rollback. For B it is re-pointing DNS back at the old cluster, which is still running. For A it is restoring the control-plane snapshot and the old packages, which is why A’s dry run must also prove the snapshot restores on a scratch node first.

B is the lower-risk path and matches the owner’s first statement of the break (fresh cluster, backup and restore). A is worth writing only if capacity for a second cluster is the blocker.

The order below is superseded by the delete-first order in the new-line simplification (SHC-1092): 7.5 moves to the front, 0.1 shrinks to a baseline reset, 3.6 merges into SHC-1058’s E1, 7.3 into 8.1, and 3.5 into SHC-1058’s D3. The steps’ content stands.

Each numbered step is one ticket and one MR. Steps inside a phase are independent unless a dependency is named.

Phase 0. The clean line.

  • 0.1 One squashed initial migration for the new line, carrying everything in 5.6 and the lease rename’s schema half.
  • 0.2 The lease rename across ce and shc: identifiers, CLI verb, API route, JSON field, error text, docs (6.1). No aliases.
  • 0.3 Node facts: node.provider, --location / --provider on init and join, terraform location / provider on shc_node and modules/core.
  • 0.4 Exclude nebula0 (any nebula*) from NIC auto-detect.
  • 0.5 Correct the stale docs (6.6).

Phase 1. Nebula as the addressing plane. Depends on 0.1.

  • 1.1 The Nebula CA in the vault; any voter signs.
  • 1.2 Nebula on from the first join; enrol over a lighthouse’s underlay address and hand out cert, CA and lighthouse list in the enrol response.
  • 1.3 The cluster plane on mesh addresses: rqlite advertise, internal API, fabric peers. The underlay and internal roles of the addressing model (its slices 2 and 3, amended).
  • 1.4 Lighthouses and relays on public-address nodes, pinned from terraform (shc_mesh, the addressing model’s slice 5); preferred_ranges per site LAN.
  • 1.5 WAN tuning from SHC-1049’s final numbers.
  • 1.6 nebula-cert staging for Proxmox nodes.

Phase 2. Fences as data. Depends on 0.3.

  • 2.1 Fence tables, the built-in and resource reconciler, shc fence ls / show.
  • 2.2 Custom fences: create, add-node, remove-node, rm, set, API, terraform shc_fence, join-time --fence.
  • 2.4 The rule store on core/scope: placement_rule, shc rule add / ls / rm, terraform shc_rule, narrowing across the cascade, and the loosening authority check per scope.
  • 2.3 shc fence explain, reading the scorer’s full Decision.

Phase 3. One placement engine. Depends on phase 2.

  • 3.1 --rule on install and update, deployfile rules:, rules on shc_deployment, and the Phase-1 admission gate that refuses an empty eligible set by name.
  • 3.2 The scorer filters by the eligible set; --node and --rule through the dispatcher; X400956NDLOCL removed.
  • 3.3 Compose and Podman relocation through the dispatcher; the MOVE-5 clamp removed.
  • 3.4 --rule on move, refusals by name naming the rule’s scope.
  • 3.5 Drain reads eligible sets from the fence store.
  • 3.6 Swarm placement keys (deploy.placement.constraints, replicas, mode: global) become rules and placement at render; a constraint with no fence form is refused by name.
  • Scheduler mechanics are SHC-1058, which consumes this phase.

Phase 4. Storage. Depends on phase 3.

  • 4.1 Pre-provision reach per volume connection (Cinder’s no-handle arm); the binder picks by target node.
  • 4.2 A move leaving a volume’s reach auto-selects the target connection.
  • 4.3 Volume failover respects the eligible set.
  • 4.4 Volumes whose reach lies outside the control-plane fence are exempt from the self-fence and from lease-expiry takeover; takeover only on a STONITH confirm or --presumed-dead (decision 24). Automatic failover waits out the self-fence before it takes over (decision 25).

Phase 5. Colocation. Depends on phase 3.

  • 5.1 placement in the socket and plug schemas, the six lint rules, and the catalog sweep that satisfies them.
  • 5.2 Relative rules: derived fences, union-find move groups, refusals by name, --rule colocate recorded on the edge.
  • 5.3 shc move --group with a write-ahead journal.
  • 5.4 External sockets: placement.location as a scored preference.

Phase 6. Control plane. Depends on phase 2.

  • 6.1 The control-plane fence; enrol forces non-voter outside it.
  • 6.2 shc cluster status reports voters against the fence.
  • 6.3 Site-loss runbook, including whether repair --force can seat a non-voter, proven on a rig.

Phase 7. The fabric, and Swarm’s retirement. Depends on phase 1.

  • 7.1 SHC-1055: fabric slice 2, nftables filtering and default-drop.
  • 7.2 SHC-1056: fabric slice 3, the Podman adapter (depends on SHC-1055).
  • 7.3 Traefik on the fabric: backends are fabric addresses from the control plane, read through the file provider; the Docker-label provider goes.
  • 7.4 Fabric MTU 1220 under Nebula.
  • 7.5 Remove the Swarm runtime, --fabric swarm, the Swarm placement renderers, the Swarm label helper, converge-swarm and drain’s Swarm reads. Depends on 7.3 and phase 3.

Phase 8. Edges and DNS. Depends on phases 3 and 7 and on SHC-1058’s cluster config directory.

  • 8.1 A Traefik on every node, fed from the cluster config directory (SHC-1058’s materialiser); the entry-point move removed.
  • 8.2 A daemon-side ACME source behind the cert broker’s seam; certificates reach every Traefik and the broker’s app deliveries from it.
  • 8.3 An edge per location fence: a proxy connection bound to a location, its pool that location’s healthy Traefik nodes; the DNS ladder picks the edge by the app’s location instead of refusing a second web connection.
  • 8.4 The per-node resolver answering mesh and fabric addresses from the materialised state, a container in the builtin Traefik project beside forward-auth (decision 26).
  • 8.5 Host and container resolver configuration forwarding the private zones to the node’s resolver; the NetBird routing-peer and nameserver-group runbook.
  • 8.6 Health-checked DNS for edges.
  • 8.7 Public DNS for a location with no edge: one healthy public node, re-pointed by the engine on failure (decision 28).

Phase 9. SSO and backups.

  • 9.1 backup.repo per fence, off-site; a mirror per location; B2 as bitspur’s second provider.
  • 9.2 Forward-auth answered from the local replica with bounded staleness.
  • 9.3 Location-loss restore runbook, proven on a rig.

Phase 10. Across. SHC-1061: the one-off migration script (6.3) and bitspur moved. promanager stays on its current line until the owner decides otherwise; the script is deleted once that is settled. A two-site proof rig (pve01 VMs and a far network with netem, the SHC-1049 shape) runs S1 to S10 end to end, including the WAN cut, and stays as a lane.

Scheduler mechanics (SHC-1058), GeoDNS, Keycloak cross-site replication, pfSense edge HA, and any stretched database.

  • Fixed by SHC-1049: docs/guide/operations/cluster-setup.md described a Nebula-based join token and a temporary Nebula during join; the Go join is LAN-token only (modules/node/commands/join.go:39-47).
  • Fixed by SHC-1049: docs/architecture/modules/nebula.md said non-raft cluster traffic flows through Nebula; nothing SHC dials does.
  • docs/guide/operations/node-rotation.md:419-449 says there is no voter rebalance; --rebalance / PromoteBeforeDeparture and shc cluster set-voter-target exist.
  • docs/architecture/cluster/network-fabric.md disagrees with itself: its header (:8-10) says slices 2 and 3 are design, its body (:35-38) says both landed, and its build inventory (:882) lists the Podman adapter as not built while ce runs podman stacks and admits them on --fabric shc (core/runtimecmd/runtimecmd.go, modules/fabric/types.go:98-106).
  • docs/architecture/cluster/network-fabric.md:1083-1098, the Nebula retirement follow-on, is withdrawn by decision 4.
  • docs/architecture/cluster/traffic-across-runtimes.md cites compose.yaml:410-412 for the published ports; they are now at ce addons/traefik/compose.yaml:732-733.
  • docs/reference/configuration/config-keys.md:540 still says split-horizon DNS was rejected; ce modules/dns/purpose.go records the 2026-08-20 re-ruling.
  • ce modules/app/dns_private.go:105-128 says the node table carries no overlay address; node.mesh_address exists.
  • ce modules/cluster/voters.go calls the unsafe escape shc cluster force_leader; the verb is shc cluster repair --force.

Superseded rather than stale, by the SHC-1058 refinements:

  • docs/architecture/cluster/network-fabric.md:262-268 accepts that compose ignores Swarm-only deploy: keys; decision 21 honours or refuses each.
  • docs/architecture/cluster/network-fabric.md:326-331,859 bakes cross-node names at render time and re-renders connected stacks on a move; decision 22 resolves them live.
  • docs/architecture/cluster/traffic-across-runtimes.md:797, “One Traefik, one node, one address”, gives way to a Traefik on every node (decision 9).