The placement-ownership fork: who schedules, Swarm or SHC?
Archived, superseded by the new line (SHC-1092), which removed Swarm and the old scheduler.
The placement-ownership fork: who schedules, Swarm or SHC?
Section titled “The placement-ownership fork: who schedules, Swarm or SHC?”★ SUPERSEDED IN PART: read
Section titled “★ SUPERSEDED IN PART: read docs/decisions/2026-08-11-rebalancer-v1-foundations.md first”docs/decisions/2026-08-11-rebalancer-v1-foundations.mdfirstThis page is the DESIGN DRAFT. The lane that built from it landed on 2026-08-11 and measured several of its claims differently. Where this page and that decision disagree, the decision is what was measured and this page is kept for its reasoning, not for its facts.
The corrections that matter here:
- R1 is RULED:
runtime: swarmis LEGACY / SECOND-CLASS. §7.1 asked it; the contention map §8 now answers it. Re-measured: 37 of 99 catalog composes plusbuiltin/traefik(the system ingress) carrydepends_onconditions Swarm cannot express; 0 Swarm-runtime stacks on either estate; 0 end-to-end verified Swarm installs, ever.- §2.4 is WRONG about the writer, and it is the correction that changes the work. It says “
SetCPUCoreshas zero callers outside generatedent/code, so the entcpu_coresfield has no writer either” and concludes the fix is “one column and one writer”. The ent BUILDER method has no callers; the COLUMN has three writers each on both axes, through raw SQL upserts: the joiner at enroll,node_self_register’s boot upsert, and the per-node fact-refresh tick.node.cpu_coreslanded 2026-07-30 andnode.memory_bytes2026-08-04. No migration was needed or added. The negative gauge §2.4 correctly diagnoses was a READ bug.- Ruling 3 of §7.2 (“reservation
quota x 1.0or limitquota x 1.5”) names multipliers that were not shipping. ce main shippedreservation = quota x 0.25andlimit = quota x 1.0. The owner’s numbers are now restored, and the swing on the commitment axis is 4x, not the 50% the row states.Everything else on this page stands, including the parts the implementation did not reach; they are named in §8 of the decision.
Status: PROPOSED. One recommendation, one owner ruling it depends on, and eight measured corrections to the ground it was decided on. Nothing here is built and nothing here should be built before the ruling in §7.1 lands.
This is lane 1 of the automatic deployment distribution / rebalancing design: the owner’s “auto distributing the deployments across different nodes, kind of like the idea of Kubernetes distributing pods across nodes.” It answers the first question, on which every other lane branches: when SHC decides that a deployment should run somewhere else, who makes that decision and who executes it?
It does not design the scheduler. It picks the lane the scheduler is built in, and states what that choice costs.
Claims are marked PROVEN (I read it in the source at the pin, or measured
it on this build host), INHERITED (asserted by
the quorum audit or
the contention map and not
re-verified here), or INFERRED (rests on documented Docker behaviour, or on
composing two proven halves that were never observed composed). Pins: shc
c0b009ba5, ce 6a3dcc5. No estate was contacted; every “live” fact
below is INHERITED from the sibling audits, never re-measured.
1. The short version
Section titled “1. The short version”Recommendation: SHC owns placement. Swarm is ratified as second-class on the runtime axis and is not the path. And the v1 scope is narrower than “a scheduler”: planned rebalancing only, node-failure failover explicitly out.
Three facts carry it, and a reader can check each one:
- The owner requires atomic. Swarm structurally cannot provide it. A Swarm
reschedule is kill the task here, start a task there. There is no point in
that sequence where anything can copy bytes and refuse. SHC’s move primitive
is atomic precisely because SHC owns the seam between stop and start:
move_health.go’s container-observation gate andmove_data_gate.go’s byte-ratio gate both sit between recover and uninstall, and both fail closed. Handing placement to Swarm means handing away the commit point, which is the one property the owner ranked as required rather than desirable. - 74 of the 99 catalogue apps are stateful and 0 of 99 carry the metadata that would make a Swarm reschedule safe for them. Swarm’s rescheduling on those apps is not a missing feature; it is an unrequested data fork (§3.3).
- The path costs a rebuild of both estates onto a runtime nobody has ever
proven. Both estates are 100% compose-runtime with zero Swarm-runtime
stacks; the deploy lane is EE-only and the CE binary compiles no
stack-deploy path at all; the fabric and the Swarm runtime are mutually
exclusive at
shc init, with no in-place flip; and 37 of 99 apps plus the system Traefik cannot bestack deploy-ed for a reason SHC cannot fix.
And the fourth, which is the reason this is cheaper than it looks: most of the SHC-side machinery is already written. §4 inventories it. The genuinely missing pieces are one dropped database column, a scoring function, a trigger, and four safety knobs the owner already named.
2. Eight corrections to the ground truth
Section titled “2. Eight corrections to the ground truth”This lane was handed a prior sweep’s findings. Six of them I confirmed. Eight I am correcting with evidence, and three of the eight change the answer materially.
2.1 M27 is half-closed: and its residue is worse than a packaging bug ★
Section titled “2.1 M27 is half-closed: and its residue is worse than a packaging bug ★”PROVEN. The contention map’s M27 (“36 of 73 catalog apps use long-form
depends_on … All are rejected before a container starts. No preflight
exists”) has been overtaken. ce/modules/app/swarm_compat.go LANDED. It
translates four constructs (long-form depends_on in its service_started
form, invalid $ interpolation, configs.<n>.content, and it names
platform: as untranslatable), and install_runner.go:1506-1514 refuses
before deployDir exists with X400073SWMXLT. The honesty branch for a
preflight that could not run has its own code, X200074SWMPRE.
But the residue is not a packaging defect; it is a Swarm capability gap, and
it is bigger than the original finding. The file refuses
condition: service_healthy and condition: service_completed_successfully
by name, and its stated reason is correct: “swarm has no start-ordering
primitive at all.” There is nothing to translate to.
Measured by me over apps/**/compose.yaml at the pin:
| count | of 99 | |
|---|---|---|
catalogue compose.yaml files | 99 | n/a |
use long-form depends_on | 47 | 47% |
use condition: service_healthy or service_completed_successfully | 37 | 37% |
declare a volumes: block | 74 | 75% |
declare x-shc-scale max: 1 | 54 | 55% |
declare x-shc-volumes | 0 | 0% |
Occurrence counts across the corpus: service_started 112,
service_completed_successfully 80, service_healthy 50.
And the system Traefik builtin is in the refused set.
ce/builtin/traefik/compose.yaml:322-324 makes traefik depend on
traefik-init with condition: service_completed_successfully, the one-shot
that writes the static config file --configFile points at. Degrading it to
list form would boot Traefik against a file that does not exist yet, which is
why the seam refuses instead.
So: 37 of 99 apps and the system ingress cannot run on the Swarm runtime, and the fix is not in SHC’s hands. It is an application re-architecture (37 packages taught not to need start ordering) or a Swarm feature that does not exist. This is the single largest cost line under option (a), and it did not exist as a line item before, because M27 read as a translation bug.
2.2 M16’s join-path half is closed
Section titled “2.2 M16’s join-path half is closed”PROVEN. The map says “ValidateSelection refuses (fabric=shc, runtime=swarm), but only at init; the join path never calls it.” The
join path calls it now: shc/modules/cluster/enroll_handler.go:329-340 runs
fabric.ValidateSelection(fabricMode, req.Runtime) against the leader’s
own posture, before any state mutation, so an illegal combination cannot
even mint a node id. It also refuses any join at all on a fabric=none
cluster (X409358FABNON). Landed in shc 3ed3ecbf1.
The render half of M16 (gate ResolveFabricRenderInput on runtime) is still
open, but it is now defence in depth rather than the only guard.
2.3 Committed-vs-free capacity is NOT missing; it is computed and published ★
Section titled “2.3 Committed-vs-free capacity is NOT missing; it is computed and published ★”PROVEN, and this is the correction that most changes the cost of option (b). The sweep recorded “committed-vs-free capacity (totals exist, nothing sums commitments)” as a missing input. The summing code exists, is wired, and runs.
ce/subsystems/impl/scheduling_source.go:197-237:
SchedulingSourceAdapter.collectScheduled walks every stack from
app.StackRepo, sums each service’s CPULimitMillicores and
MemoryLimitBytes, and buckets the total per node by the deployment row’s
node (Swarm-runtime stacks aggregate cluster-wide into a separate bucket
deliberately, because “scheduled on node N” does not apply there).
ce/subsystems/impl/collectors.go:339-344 wires
metcoll.NewSchedulingCollector(svc, SchedulingSourceAdapter{repo, Pool})
into the collector list unconditionally, not behind the quota edition
seam, not behind a flag. It is leader-only and emits ten shc_node_* gauges
including shc_node_cpu_scheduled_millicores,
shc_node_cpu_schedulable_millicores and
shc_node_cpu_schedule_utilization_ratio.
The scheduler’s fitness function has been running as a Prometheus gauge. What is missing is not the input. It is a consumer: no placement decision reads it.
2.4 …but the capacity half is structurally zero, so the gauge is wrong ★
Section titled “2.4 …but the capacity half is structurally zero, so the gauge is wrong ★”PROVEN from code; NOT verified on an estate. ListNodes
(scheduling_source.go:150-193) scans exactly three columns:
SELECT id, hostname, runtime FROM node WHERE cluster_state IS NULL OR cluster_state IN ('healthy', 'degraded')and its own comment says why:
role and per-node resources are NOT columns on
node(the 2026-07-22 re-flatten dropped both; neither had a writer, so live rows always read NULL …). Role defaults to “worker” and Resources stays the zero capacity … a future resource-aware scheduler re-adds the column together with a writer.
SetCPUCores has zero callers outside generated ent/ code, so the ent
cpu_cores field has no writer either. The synthetic runtime-probe fallback
(NumCPU / /proc/meminfo / statfs) fires only when the node table is
empty: a fresh pre-bootstrap install. On any real multi-node estate the DB
rows exist, so the fallback never runs and every node reports zero capacity.
Consequence, read straight off collectors/scheduling.go:63-76:
shc_node_cpu_capacity_millicores= 0 for every node;shc_node_cpu_scheduled_millicores= a real, non-zero sum;shc_node_cpu_schedulable_millicores=0 − scheduled= negative;- both
*_schedule_utilization_ratiogauges are never emitted, because they are guarded bycapacity > 0.
This is a defect in its own right and worth booking regardless of the fork: a published gauge whose value is structurally negative. It is also the precise shape of the missing work: one column and one writer, and the code already reserves the shape.
2.5 The placement candidate-set filter already exists, and it is better than pickTarget
Section titled “2.5 The placement candidate-set filter already exists, and it is better than pickTarget”PROVEN. The sweep named pickTarget in shc/modules/cluster/drain.go:512
as “the only node-scoring code in the product”, accurate for scoring, but
the candidate-set half exists elsewhere and is stronger. ListNodes uses a
positive allowlist (cluster_state IS NULL OR IN ('healthy','degraded'))
with a recorded rationale: it was switched from a negative != 'removed'
filter “so a new quarantine state is excluded by default rather than silently
admitted”, and it excludes both unreachable and lease_pending.
pickTarget by comparison filters on availability=active only, and sorts on
Count, the number of running deployment rows, tie-broken by node id,
preferring a runtime match. A scheduler wants the allowlist from ListNodes
and the ordering from a real fitness function; neither is pickTarget’s
row-count.
2.6 Node-failure-triggered relocation with fencing already exists, for volumes
Section titled “2.6 Node-failure-triggered relocation with fencing already exists, for volumes”PROVEN. ce/subsystems/impl/volume_failover.go is a leader-only automatic
failover-rebind ticker with a written safety contract:
- off by default (
storage.volumes.auto_failover), and flag-off no-ops as the first statement, so behaviour is exactly operator-driven rebind; - no move without a force-detach proof: either the old node’s attachment
lease is provably expired (
leased_until < now) and a ≥45s dwell has elapsed (lease TTL 15s ≪ dwell, so the node-local lease-loss stop has already had time to stop, unmount and flush), orStonithAgent.PowerOff(node)confirms. The defaultNoopStonithAgentnever confirms;modules/storage/providers/hcloud/stonith.gois a real one; - one poll signal never drives a detach: dual-signal plus dwell;
- never crosses an AZ (
node.labels["az"]); - the lease bump is a single monotonic writer and runs before attach.
node.ClusterStateLeasePending exists as a distinct state from unreachable
precisely so a quarantine cannot be mistaken for an authorization, and its
doc comment says it “is excluded from the placement candidate set”.
It is scoped to RWO block volumes, and 0 of 99 apps declare
x-shc-volumes, so it fires for nothing today. But the statement “SHC would
have to build node-failure detection and fencing” is wrong: the detection,
the quarantine vocabulary, the dwell discipline, the STONITH seam and one
provider implementation are written, unit-tested and shipped.
2.7 shc node drain already performs real relocations through the move orchestrator
Section titled “2.7 shc node drain already performs real relocations through the move orchestrator”PROVEN. ce/daemon/services_extra.go:2184-2260:
clusterDeploymentMigrator.Migrate calls clone.Service.Move with the same
tenant, stack and environment and only the node changed: “the same code path
as shc move <stack> <stack> --node <target>, so a drained workload gets the
backup, the cross-node ship, the bring-up on the target, and the source
teardown.” Errors are the caller’s verdict: drain records a per-stack failure
and never counts the row migrated.
Its comment also records what it replaced, which is worth quoting because it is the failure mode any rebalancer will be tempted into:
It used to be
UPDATE deployment SET host_node = ?. Nothing moved: the containers kept running on the node being drained while the deployment table claimed otherwise … That is worse than doing nothing.
So an end-to-end, operator-triggered, journalled, gated, one-at-a-time scheduler-of-one already ships. A rebalancer is that loop with a different trigger and a better target function.
2.8 The RWO single-writer clamp fires for nothing
Section titled “2.8 The RWO single-writer clamp fires for nothing”PROVEN. ce/modules/app/swarm_placement.go:104: ApplyVolumePlacement
returns the compose byte-identical when len(placements) == 0, and placements
come from x-shc-volumes declarations. Zero of 99 apps declare one. The
contention map lists the RWO clamp under “what holds” and calls it “the model
to generalise”, correct as a design, but it currently protects nothing, and
that matters enormously for option (a) (§3.3).
3. Option (a): let Swarm do it
Section titled “3. Option (a): let Swarm do it”3.1 What SHC would have to start doing
Section titled “3.1 What SHC would have to start doing”| # | new obligation | why | blocked on |
|---|---|---|---|
| a1 | write deploy.resources.reservations | Swarm schedules on reservations; SHC writes only limits, which are a post-placement cgroup setting. Swarm’s scheduler has nothing to honour today | R2, which the map says explicitly: “No code should be written before this is ruled” |
| a2 | write spread / anti-affinity constraints | PlacementConstraint is a drain-time filter, not a render input | none |
| a3 | fix A1 first | on every Swarm stack, pinned or not, the first shc update renders node.hostname==node-<hex> (a hostname no Docker node has) and the task goes Pending forever | A persist-and-replay fix was attempted and REFUTED: it broke shc recover and the presumed-dead move on every runtime and turned shc move on Swarm into a silent no-op |
| a4 | tolerate Swarm rescheduling | means giving up the commit point, §3.3 | the owner’s atomic is required |
| a5 | build M-EVENTS, M-GENERATION, M-TASKID, M-PREFLIGHT | SHC currently learns nothing about what Swarm does; the dominant class in the contention map is BLIND, not FIGHT | none |
| a6 | re-architect 37 apps + the system Traefik | they need start ordering; Swarm has no start-ordering primitive | not fixable in SHC (§2.1) |
3.2 What it inherits
Section titled “3.2 What it inherits”The whole contention map: 36 rows where both control planes act, of which
M1–M8 can lose or corrupt data. Plus the quorum audit’s A1–A7 and D1–D7.
Plus the fact that R1 is UNRULED: “Is runtime: swarm a supported
production configuration, or is it de facto legacy?” If the owner rules
“supported”, M27 is a P0 by the map’s own words, and §2.1 shows the P0 is
larger and less fixable than the map recorded.
Three inherited items deserve naming because they are structural, not backlog:
- Swarm is not opt-in per deployment.
ensureSwarmAttachableNetworksgates on host Swarm state, not the stack’s runtime, so on any Swarm manager SHC’s shared networks become overlays in the Swarm raft, for compose stacks. (INHERITED, proven live on promanager/hel1 by the sibling audits: twelve overlay-scope shared networks while every deployment is compose anddocker stack lsis empty.) The decision is not “should SHC keep Swarm support” but “should SHC nodes join a Swarm at all.” - The swarm raft is a second consensus store SHC does not manage. It initialises it once, hands every joiner the worker token, never promotes, never demotes, never reads the manager roster, and has no recovery verb. Both estates run exactly one manager: quorum 1 of 1, survives 0 losses. (INHERITED.)
- The fabric and the Swarm runtime are mutually exclusive at
shc init, with no in-place flip. PROVEN:ce/modules/fabric/types.go:93-108refuses(fabric=shc, runtime=swarm), and it is enforced at init (modules/cluster/commands/init.go:182,modules/cluster/service.go:813) and now at join (§2.2).
3.3 The honest statement about stateful stacks ★
Section titled “3.3 The honest statement about stateful stacks ★”Swarm reschedules by starting a task somewhere else. The data does not follow. Stated plainly, because it is the decisive fact and it is easy to soften.
The chain, with each link’s evidence class:
- An unbound DATA mount renders as a per-node named volume, a host bind
under
<deployDir>/mountsdoes not exist on a remote worker (install_runner.go:3066-3090). INHERITED (contention map M1; the quorum audit reached the same branch independently). - On reschedule, Docker provisions a fresh, empty volume wherever the task lands. INFERRED from documented Docker behaviour.
- The app comes up blank, not broken, not down. It serves. Users authenticate against an empty database and start writing.
- This is not data loss; it is a data fork. The real bytes are intact on the old node. A second, authoritative-looking copy is now accumulating writes. Nothing in the product reconciles two.
- Then the next backup resolves “the mounting service’s most recent task node” and archives whatever volume is there (the empty one) and reports success; retention then rotates the last good archive away. INHERITED (M2); the map marks the composition of M1 and M2 as INFERRED because it was never observed end to end. That composition is what turns the fork into loss.
- The guard that would prevent all of it (the RWO pin-plus-
replicas: 1clamp) fires for zero of 99 apps (§2.8). - 74 of 99 apps declare a
volumes:block.
And none of it requires anyone to ask for anything. An unpinned Swarm service reschedules because that is what Swarm does. There is no per-deployment opt-in.
The RWO clamp is the right shape (SHC stating an invariant in Swarm’s own vocabulary rather than fighting the scheduler) but generalising it to the whole catalogue means declaring volume metadata for 74 apps and pinning them, at which point Swarm’s scheduler is disabled for three quarters of the catalogue and you have paid the entire contention map for the remaining quarter.
3.4 What Swarm genuinely gives, stated fairly
Section titled “3.4 What Swarm genuinely gives, stated fairly”One thing, and it is real: automatic restart of a task on a surviving node after a node dies, with no operator action. SHC has no equivalent; on compose, a dead node’s deployments stay dead until someone acts.
For a stateless service (an app with no volumes: block, 25 of 99, and
predominantly proxies, fixtures and front-ends) that is genuine, free HA. It
is exactly the capability §5.2 says should be reached for later by a different
route, and it is the one line in Swarm’s favour that this document does not
argue against.
4. Option (b): SHC does it
Section titled “4. Option (b): SHC does it”4.1 What already exists
Section titled “4.1 What already exists”| capability | where | state |
|---|---|---|
| atomic move, journalled | ce/modules/clone/move_journal.go, durable node-local journal, checksum envelope, tmp+fsync+rename, phases written AHEAD of the act; deliberately a file, not a table, so it needs no migration | BUILT |
| crash recovery | ce/subsystems/impl/move_journal_recovery.go, priority 73, deps {container_runtime, ent_client}, ModeAny not leader-only (the killed daemon is the one whose disk holds the record, and it is usually not the leader), returns immediately and works in a goroutine | BUILT, ★ daemon-start only, no timer |
| the commit point | move_health.go verifyPhase (container observation, 180s bounded poll, fail-closed) and move_data_gate.go dataPhase (zero bytes always refuses; ratio < 0.25 refuses; unmeasurable refuses), two gates between recover and uninstall | BUILT |
| real relocation execution | clusterDeploymentMigrator.Migrate → clone.Service.Move (§2.7) | BUILT, wired to drain |
| candidate-set filter | ListNodes positive allowlist excluding unreachable + lease_pending (§2.5) | BUILT |
| committed-resource accounting | collectScheduled, per node, cluster-wide, on the leader (§2.3) | BUILT, wired, no consumer |
| node quarantine vocabulary | ClusterStateLeasePending / Unreachable, distinct by design | BUILT |
| fencing discipline | dwell + lease-expiry proof + STONITH seam + real hcloud STONITH agent + same-AZ rule (§2.6) | BUILT, off by default, fires for nothing |
| multi-stack scope | --closure flag, BFS, capped at 8 stacks, closes the --all-connections one-hop blocker | BUILT |
| storage move mechanism | three-strategy selector, fail-closed classifier, fenced block-volume binder with explicit Rebind, Hetzner + 8 other providers | BUILT, no declaration and no consumer |
4.2 What is genuinely missing
Section titled “4.2 What is genuinely missing”Ordered by whether it is plumbing or a decision.
Plumbing (small, and the code reserves the shape):
- A node capacity column and its writer. Dropped 2026-07-22 for want of a writer; the comment already names re-adding it “together with a writer” as the resource-aware-scheduler prerequisite (§2.4). Fixes the negative gauge as a side effect.
- A per-node committed-disk figure. Capacity-only today; there is no disk
analogue of
collectScheduled.quota.ServiceOptions.Pathsis never set in production wiring, soScanDiskwrites 0 forever, and that bug is runtime-independent (INHERITED, M32). - A consumer for §2.3’s numbers: a scoring function that reads them instead of a Prometheus exporter.
Decisions (cannot be plumbed around, §7):
- Commit against reservation (
quota × 1.0) or limit (quota × 1.5)? The accounting today sums limits; the owner’s model says the reservation is the commitment. - Compare against total or free? The three physical gates in
shc/modules/quota/enforcement.goare well written, unit-tested, have zero non-test callers (PROVEN, every hit isenforcement_test.go), and compare against total. Open indocs/decisions/2026-08-04-physical-capacity-gates.md. They would also need a capacity source that exists, which is item 1. - The four knobs the owner already named: a per-deployment never-move annotation, a concurrency cap, an audit trail answering why did it move this, and a kill switch: “cheap if designed in, expensive if bolted on.”
- Per-app STANDBY declaration. The background-worker trap: Sidekiq, Nextcloud cron and Windmill workers write without receiving any HTTP request, so “start the target fully, then flip” double-runs cron against the same database. The answer is a three-state cutover with a per-app standby mode, and standby must be declared, per app, by the packager.
Not missing, and worth saying so: cross-node live utilization. StatsCache
is per-process, 15s, and empty before its first tick. A v1 scheduler should
schedule on declared resources (the quota-derived reservation) and not
on observed load. That is the honest v1 and it should be stated rather than
hidden behind an intent to add live signals later.
4.3 What SHC would have to build that Swarm gives free
Section titled “4.3 What SHC would have to build that Swarm gives free”Node-failure detection and rescheduling. §5.2 argues this should be out of v1, and §2.6 shows the hard half is already written.
5. Option (c): the third paths
Section titled “5. Option (c): the third paths”5.1 Two the repo half-considers, and why both fail
Section titled “5.1 Two the repo half-considers, and why both fail”(c1) SHC schedules, Swarm executes. SHC computes placement and stamps
deploy.placement.constraints on every service; Swarm becomes a dumb executor.
Cost: you inherit all 36 contention rows and the second raft, and you have
disabled the only thing Swarm was for. You still cannot stack deploy 37 of 99
apps or the system ingress (§2.1). You still have A1 wedging every stack on
first update. Reject: maximum inherited cost, zero inherited benefit.
(c2) Swarm for stateless, SHC for stateful. Runtime is already a per-deployment column, so this is expressible without new vocabulary.
Cost: both control planes live simultaneously, so every FIGHT row is active
rather than latent, and the classifier that decides which is which
disagrees with itself in five places and fails open to compose in three
(INHERITED, §3.1 of the map). The stateless set is 25 of 99 and is mostly
proxies and fixtures. Podman nodes cannot join a Swarm at all
(ValidateSelection refuses (fabric=swarm, runtime=podman)). And the fabric
is refused on any node that would carry a Swarm-runtime stack, so a mixed
estate cannot have the fabric, meaning the strategic data plane is
unavailable to the 74 stateful apps in exchange for HA on the 25 that matter
least. Reject: buys the cheapest quarter of the catalogue at the price of
the whole map.
5.2 The third option that matters: split the capability the question conflates ★
Section titled “5.2 The third option that matters: split the capability the question conflates ★”“Who owns placement” bundles two capabilities with opposite risk profiles, opposite feasibility, and, critically, opposite rank in the owner’s own constraint.
| rebalancing | failover | |
|---|---|---|
| trigger | policy: this node is better | a node died |
| source node | alive | gone |
| atomic? | yes: stop → ship → verify → gate → uninstall | impossible: you cannot copy bytes off a dead node |
| honest answers | one: move it properly | three: shared storage, a live replica, or it stays down |
| owner’s rank | ”Atomic is required" | "HA is desirable but optional” |
| Swarm | cannot do it (no seam) | does it, destructively for 74/99 apps |
| SHC | already does it (drain, §2.7) | does not, but owns the fencing discipline (§2.6) |
Read the row that matters: Swarm provides only the optional capability, and provides it in the form that is most dangerous for three quarters of the catalogue. SHC already provides the required one.
The consequence for scope is large and good: v1 needs no node-failure detection at all. That is the expensive, dangerous half, the half where a network partition becomes a split brain and where “the node is unreachable” and “the node is dead” must be distinguished under time pressure. Cutting it from v1 removes the entire fencing-correctness burden from the first release, and leaves it to be reached later along a route that already has quarantine states, a dwell, a lease-expiry proof, a STONITH seam and a real hcloud STONITH agent written and shipped off-by-default.
The repo does not consider this split because both sibling documents ask “is Swarm supported?” rather than “which of the two things Swarm does are we actually buying?“.
6. Recommendation
Section titled “6. Recommendation”Take (b), scoped by (c3): SHC owns placement; v1 is planned rebalancing only on the compose runtime; node-failure failover is explicitly out of v1 and is a later, separately-gated capability.
Concretely:
- Answer R1 as “de facto legacy / second-class, with the limitations written down.” The contention map’s §5 is already drafted as that statement. This recommendation depends on that answer (§7.1).
- Build the scheduler on the compose runtime, on the fabric path. The fabric is the strategic multi-node data plane, it is far past scaffold, and it is what both estates can actually reach.
- v1 trigger set: operator-invoked (
shc rebalance, plan-then-apply) and drain. No autonomous timer in v1; the owner’s kill switch is cheapest when the loop does not yet run itself. - v1 scoring: declared reservations against real node capacity, once the dropped column has a writer. Not live utilization.
- Failover stays out, and the product says so: a dead node’s deployments stay down until the node returns or an operator moves them. That is a worse guarantee than Swarm’s on paper, and a better one in practice for 74 of 99 apps, because Swarm’s version silently forks their data.
6.1 What this recommendation does not give you
Section titled “6.1 What this recommendation does not give you”Stated plainly, because the house standard is that a proposal names what it cannot do:
- No automatic recovery from node death in v1. This is a real regression against what a Swarm estate would nominally offer, and it must be documented, not glossed.
- The fabric’s move ripple. Alias→IP bindings are fixed at render time, so
moving a stack triggers a re-render of its connected dependents
(
extra_hostschanged). The fabric design names this as “this design’s main UX regression versus Swarm DNS” and accepts it for v1, but a rebalancer makes moves routine rather than rare, which changes how much that regression costs. This is the single most under-weighted cost of the recommended path and lane 2 or 3 should size it before anything is built. - No live-load awareness. v1 packs declared reservations. A node whose declared commitments are modest and whose actual CPU is pinned will look empty.
- No disk dimension. There is no per-node committed-disk figure anywhere (§4.2 item 2), so v1 cannot avoid filling a disk.
- It strands the Swarm code that exists. If R1 rules “legacy”, the contention map’s engineering-choices list should mostly not be worked: it should be reduced to making the refusals honest and loud. That is a real cost: a body of correct, well-reasoned work stops being on anyone’s path.
- It does not resolve the storage fork. The block-volume mechanism is built
with no declaration and no consumer;
Volume.access_modeis constrained torwo|rox, so a shared filesystem can never own a volume row and both shared-FS classifier arms are dead code. A shared-storage answer would change the failover calculus completely, and it is outside this lane.
7. What needs an OWNER RULING
Section titled “7. What needs an OWNER RULING”7.1 The ruling this recommendation depends on
Section titled “7.1 The ruling this recommendation depends on”Is runtime: swarm a supported production configuration, or de facto
legacy? (The contention map’s R1, restated because this lane cannot be
decided without it.)
- If legacy / second-class: take this recommendation. The map becomes documentation, the fabric answers multi-node, and the 36 rows stop being backlog.
- If supported: this recommendation is wrong, and the first thing owed is not a scheduler; it is §2.1. The system ingress and 37 of 99 apps cannot run on the runtime, for a reason SHC cannot engineer around, and shipping a scheduler onto a runtime that cannot host the catalogue would be building the second floor first.
7.2 The rest
Section titled “7.2 The rest”| # | ruling | why it cannot be an engineering choice |
|---|---|---|
| 2 | Does the node table get its capacity columns back, with a writer? | It was deliberately dropped as unwritten; re-adding it is a schema decision and the migration doctrine applies (make migrate, never flatten) |
| 3 | Does the scheduler commit against reservation (quota × 1.0) or limit (quota × 1.5)? | The accounting sums limits today; the quota model says the reservation is the commitment. Both are defensible; the difference is a 50% swing in effective cluster capacity |
| 4 | Total or free? The three physical capacity gates | Already open in docs/decisions/2026-08-04-physical-capacity-gates.md. Under (b) the answer is pulled hard toward free, but it remains the owner’s |
| 5 | Is rebalancing opt-in per estate, per tenant, or per deployment, and what is the kill switch’s shape? | This is the knob the owner said is cheap now and expensive later |
| 6 | Does the fabric become mandatory for multi-node? | ValidateSelection already refuses a join on a fabric=none cluster, so the product half-assumes it. Saying it makes the fork explicit at shc init |
| 7 | Who declares standby: the packager, per app, in app.yaml? And what does SHC do with an app that declares none? | Refuse to move it, or move it and accept a double-run window. Both are product statements |
| 8 | Is there a shared-storage answer on the roadmap? | It would change §5.2’s failover row from “impossible” to “possible”, which would reopen this fork |
8. Rejected alternatives, with their costs
Section titled “8. Rejected alternatives, with their costs”| rejected | cost of taking it |
|---|---|
| (a) Let Swarm do it | A rebuild of both estates onto a runtime never verified end to end, chosen at shc init with no in-place flip; 37/99 apps + the system ingress unrunnable for a reason SHC cannot fix; the atomic commit point given up, contradicting the owner’s one required property; an unrequested data fork on 74/99 apps with a backup path that converts it to loss; the entire 36-row contention map and the sibling’s 14 defects inherited; R2 blocking any code at all |
| (c1) SHC schedules, Swarm executes | Every cost of (a) except Swarm’s rescheduling, in exchange for none of Swarm’s benefit |
| (c2) Swarm stateless / SHC stateful | Both control planes live at once (all FIGHT rows active, not latent) via a classifier that fails open to compose in three of five sites; the fabric unavailable to the estate; podman excluded; and the split buys HA for the 25 apps that need it least |
| (b) with failover in v1 | The full fencing-correctness burden (partition vs death, split brain, STONITH as a hard dependency) on the first release, for a capability the owner ranked optional, and on top of a scheduler that has never run |
| (b) with an autonomous timer in v1 | A loop that moves production workloads on its own before anyone has watched it choose once. The kill switch becomes load-bearing on day one instead of being a safety net |
| Do nothing; keep install-time placement | Placement stays “whoever handled the request” forever (AnchorIntent writes the local node; PlacementIntent writes a pin only from an explicit --node). Capacity drift is invisible, and shc_node_cpu_schedulable_millicores keeps publishing a negative number |
9. Evidence
Section titled “9. Evidence”PROVEN: read at the pin, or measured on this build host. The catalogue
counts in §2.1 (99 files; 47 long-form depends_on; 37 carrying
service_healthy / service_completed_successfully; 74 with a volumes:
block; 54 with x-shc-scale max: 1; 0 with x-shc-volumes), by a textual
scan: the compose files are Jinja templates and 53 of 99 do not parse as YAML,
so a YAML-based count under-reports by roughly half and should not be trusted.
ce/builtin/traefik/compose.yaml:322-324. swarm_compat.go’s refusal set and
its wiring at install_runner.go:1506-1514. enroll_handler.go:329-340.
scheduling_source.go:150-237 and collectors.go:339-344 and
collectors/scheduling.go:63-76. drain.go:512 pickTarget.
services_extra.go:2184-2260. volume_failover.go’s safety contract and
node/types.go:79-101. swarm_placement.go:104. fabric/types.go:93-108
with its three call sites. runtime_gate.go (CE compiles no stack-deploy
lane; X403977SWMEEO). move_journal_recovery.go’s ModeAny rationale and
priority. Zero non-test callers for EnforceNodeCapacity,
EnforceSwarmPoolCapacity, EnforceLargestReplicaFitsNode, and for
SetCPUCores outside generated ent/.
INHERITED: asserted by the sibling audits, not re-verified here. Every
live-estate fact: three nodes each, one Swarm manager each, empty
docker stack ls on both, twelve overlay-scope shared networks on
promanager/hel1, engine 26.1.5 / compose 2.26.1. The full contention map
M1–M36 and the quorum audit’s A1–A7 / D1–D7 except where §2 corrects them. The
compose-verb exit-code table. That docker compose up over a Swarm-shaped file
creates placement-free duplicates.
INFERRED. That a rescheduled task gets a fresh empty volume (documented
Docker behaviour). The §3.3 chain composed end to end; the map marks the
M1→M2 composition as never observed, and I did not observe it either. That
shc_node_cpu_schedulable_millicores in fact reads negative on a live estate:
the arithmetic and the absent writer are proven from code, the value was not
read from an estate, because this lane made no estate contact.
Owed, and not doable in this lane. Read the scheduling gauge family off a live estate and confirm the negative (§2.4), the cheapest confirmation here and worth doing before anything is built. Size the fabric’s move-ripple cost on a real dependent graph (§6.1). Both belong to whichever lane comes next; a disposable pair is enough for the first and a static analysis of the catalogue’s integration edges is enough for the second.
References
Section titled “References”- Swarm quorum, and status/health under Swarm, the two rafts, the observation tier, five rulings; §2.2 corrects its M16 sibling
- The swarm contention map, 36 rows, seven rulings; §2.1 corrects M27, §2.8 qualifies the RWO clamp
- Network fabric (design), the strategic data plane; its “Move” section owns the ripple cost in §6.1
- Deployment binding model (proposal),
bound-ness, the data-loss refusal, the
UNANCHOREDcell - Node addressing model (proposal),
owns
bindand the terraform-boundary removal mechanism - Physical capacity gates (decision), the total-vs-free question, still open, now on this lane’s critical path
- The trilemma, giving up the scheduler is a coherent pick; making it accidentally is not