The capacity substrate: what a rebalancer has to be able to read
Status: open design. Built in part on 2026-08-11, and the decision named below says which claims changed. The other steps are not built.
★ SUPERSEDED IN PART: read
Section titled “★ SUPERSEDED IN PART: read docs/decisions/2026-08-11-rebalancer-v1-foundations.md first”docs/decisions/2026-08-11-rebalancer-v1-foundations.mdfirstThis page is the DESIGN DRAFT. The lane that built from it landed on 2026-08-11 and measured several of its claims differently. Where this page and that decision disagree, the decision is what was measured and this page is kept for its reasoning, not for its facts.
The corrections that matter here:
- §9.1 is ANSWERED. The reservation multiplier is 1.0 and the limit multiplier 1.5, the owner’s model. This page was written to be correct under either value and it is; the value is now 1.0, so a node’s committed sum is 4x what it was, and the live estates read ~69% committed rather than ~17%.
- §1’s Q1 row and §5.4 both understate what already exists. The node capacity columns have three writers each and had them before this page was drafted; the defect was that
scheduling_source.gonever scanned them. Fixed with no migration.- §5.4’s fourth gate (
EnforceNodeHeadroom) is NOT built, and §9.3 is the reason: its numbers need the daemon’s own footprint measured on both estates first, and an absolute floor invented rather than measured is a refusal threshold nobody can defend. The PROPOSED placeholder of 1000m / 2 GiB was deliberately not shipped.- Steps 0, 1, 4 and 5 of §6 are NOT built. The utilization axis, the free-disk fact, the node-addressed free-space read and the durable digest all remain owed. Step 2 (the commitment reader) and the gate half of step 3 are built; the ce-side
CapacityAdvisorseam is not; what landed instead is a PLACEMENT seam on the same install path.Everything else on this page stands, including the parts the implementation did not reach; they are named in §8 of the decision.
As written: PROPOSED. Design only; nothing on this page is built. Steps 0-2 of §6 are safe to build before any ruling; step 3 is blocked on §9.1 and §9.3. This is lane 2 of the automatic-distribution design: not the scheduler, not the move, but the layer underneath both: what does node N have, and what is already promised on it. A scheduler cannot exist without it and a rebalancer that guesses at it is a random walk with a commit point.
Baselines. shc main c0b009ba5, ce main 6a3dcc5. Every claim tagged
MEASURED was read out of those two trees during this session and the file
and line are given. Claims tagged PROPOSED, ASSUMED or RULING OWED
are exactly that. Claims taken from another page without re-verification are
tagged INHERITED and name their source.
0. Four corrections to the ground truth this lane was handed
Section titled “0. Four corrections to the ground truth this lane was handed”The brief for this lane carried four statements that the trees do not support. Three of them make the job smaller than it looked; one makes it larger. All four change the design, so they come first.
0.1 Cross-node live utilization already exists, and it is already wired
Section titled “0.1 Cross-node live utilization already exists, and it is already wired”The brief said the StatsCache is “per-process, 15s, empty before the first
health tick, and a consumer on another node cannot read it at all.” The first
three clauses hold. The fourth does not.
MEASURED. ce modules/metrics/collectors/peer_nodes.go is
PeerNodeStatsCollector: a fan-in that asks every peer for its local
StatsCache view over the cluster mTLS internal API and merges the result into
the caller’s own cache. It is:
- wired:
ce subsystems/impl/collectors.go:368, unconditionally, in the collectors subsystem’s start path; - running on every node, not just the leader;
LeaderOnly()returnsfalseand the comment says why (“every node aggregates its own cluster-wide view so the HUD reflects all nodes no matter which node it connects to”); - served by a real endpoint:
GET /internal/_node_stats, registered atce modules/metrics/router.go:33, returning{"nodes":{…},"containers":{…}}; - dialled over the mesh peer client:
ce subsystems/impl/peer_node_stats_adapter.go, peers enumerated from the node registry byinternal_address, local node excluded; - ticking at 2 seconds;
resolveTickInterval(collectors.go:703) readstiming.tick_interval, whose shipped value is"every 2 seconds"(ce core/config/default.yaml:244) and whose unparseable fallback is 2s.
So the transport question for (b) is not “build one”. It is “the one that exists carries three defects that make it unsafe for a scheduler to read”: §3.
0.2 A per-node disk fact exists too, and it is the wrong number, measured six ways
Section titled “0.2 A per-node disk fact exists too, and it is the wrong number, measured six ways”The brief said “per-node free disk; no fact exists anywhere.”
MEASURED. NodeStats (ce modules/metrics/stats_cache.go:32-34) carries
DiskTotalBytes, DiskUsedBytes and DiskPercent, produced by
readDiskStats(s.statfsPath) where statfsPath defaults to /var/lib/shc
and falls back to / when that does not exist
(ce modules/metrics/collectors/system_tick_source.go:92-97). It rides the
fan-in in §0.1. It reaches every node’s cache every 2 seconds.
What does not exist is a usable free figure. readDiskStats computes
used = total − Bfree (system_tick_source.go:504-517), so any consumer deriving
free = total − used recovers Bfree, which counts the root reserve, the
5% of an ext4 filesystem an unprivileged writer cannot touch. The two places
in the product that actually gate on free space both use Bavail, and say
in their own comments that the root reserve is not budget.
MEASURED: six syscall.Statfs call sites, three definitions of “free”:
| site | path | field |
|---|---|---|
ce modules/clone/preflight_space.go:134 DiskFreeBytes | staging dir, walks to nearest existing ancestor | Bavail |
ce modules/backup/preflight_space.go:169 diskFreeBytes | backups dir, same ancestor walk | Bavail, a byte-identical second copy of the function above |
ce modules/metrics/collectors/system_tick_source.go:504 readDiskStats | /var/lib/shc || / | Bfree |
ce modules/health/internal_router.go:288 | /, hardcoded | Bfree |
ce modules/health/maintenance.go:150 defaultDiskUsageProbe | caller-supplied | Bfree |
ce daemon/services_seams_maint.go:317 maintDiskUsageProbe | caller-supplied | Bfree, a second copy of the probe above |
Two of the six are literal duplicates of another two. Nothing reconciles them. A node’s disk pressure therefore reads differently depending on which surface asked, and the two surfaces that refuse work disagree with the four that report health.
0.3 The reservation is already summed per node, and the sum is wrong three ways
Section titled “0.3 The reservation is already summed per node, and the sum is wrong three ways”The brief said “nothing sums commitments.”
MEASURED. ce subsystems/impl/collectors.go:1046-1072
(aggregateSourceAdapter.Stacks) walks every stack, sums
sm.CPURequestMillicores and sm.MemoryRequestBytes across its services, and
labels the row with a NodeID. Those become the OTel gauges
shc_stack_cpu_request_millicores / shc_stack_memory_request_bytes
(ce modules/metrics/collectors/aggregate.go:142), and the per-service
variants carry node_id too (aggregate.go:116).
It is not a commitment ledger, for three independent reasons:
- It does not multiply by replicas.
row.CPURequest += float64(sm.CPURequestMillicores); nodesired_replicasfactor, while the quota charge on the very same table does multiply (ce modules/app/quota_render_charge.go:73,quota × desired_replicas). Two summations of one table disagree on the replica factor and the metric one is the wrong one. - Its node attribution is the routing scalar.
NodeIDcomes froms.Node(deployment.recorded_node_id) or, when a container row lands first, from that container’s node.recorded_node_idnames whoever handled the install request; under swarm it is not even a member of the set of nodes running the tasks. That is the exact confusion thedeployment_nodeset was created to end (ce ent/schema/deployment_node.go, whole header). - Nothing in the product can read it back.
ce modules/metrics/router.gostates it plainly: “Per B9 there is NO /metrics scrape endpoint; push-only via the OTLP exporters”. The numbers leave for a telemetry backend and never return.
So the ingredients for a commitment ledger are all persisted and one wrong aggregation of them already ships. That makes §2 a smaller job than it looked and a more delicate one: the failure mode is not an empty page, it is a plausible number.
0.4 The reservation multiplier is contested, and it moves every figure on this page by 4×
Section titled “0.4 The reservation multiplier is contested, and it moves every figure on this page by 4×”This is the correction that makes the job larger.
The brief stated the owner’s model as reservation = quota × 1.0 and
limit = quota × 1.5. MEASURED, ce main disagrees on both constants:
defaultReservationRatio = 0.25:ce modules/app/resource_reservations.go:102defaultLimitRatio = 1.0:ce modules/app/resource_quota.go:174- and the chain in the shipped header (
resource_quota.go:12-14) islimit = quota × limit_ratio/reservation = quota × reservation_ratio, i.e. the owner’s shape with different values.
INHERITED (memory quota_derivation_model, owner 2026-08-03): the owner
stated reservation_multiplier default 1.0 and limit_multiplier default 1.5,
and explicitly rejected deriving the reservation from an app-authored limit.
The shipped code derives from the quota (the shape objection is resolved),
but ships 0.25 and 1.0.
Why this lane cares more than the quota lane does. The quota charge is the quota, always, so the ratios do not move the bill. The commitment ledger’s operand is the reservation, so:
| committed sum, per unit of granted quota | |
|---|---|
| ratio 0.25 (shipped) | 25% |
| ratio 1.0 (owner’s stated model) | 100% |
At 0.25 a node is “full” only when four times its RAM has been sold on it. At 1.0 it is full when its RAM has been sold once. Every threshold, every headroom number and the entire question of whether a capacity gate ever fires turns on this. RULING OWED (§9.1). This page is written so the ledger is correct under either value, and it does not pick one.
1. The four questions the substrate has to answer
Section titled “1. The four questions the substrate has to answer”Everything below is in service of exactly these. A consumer that needs a fifth thing is out of scope for this lane.
| # | question | answered from | today |
|---|---|---|---|
| Q1 | how big is node N | node.cpu_cores, node.memory_bytes | exists, NULL = unknown, three writers |
| Q2 | how much of N is already promised | Σ reservation × replicas over what occupies N | does not exist (§0.3 is not it) |
| Q3 | how loaded is N right now | NodeStats via the peer fan-in | exists, unsafe to read (§3) |
| Q4 | how much writable disk does N have, per path-class | statfs | exists, wrong field, one path (§0.2) |
★ Q1 and Q2 are claims about intent and are answerable from replicated tables at any time, including while a node is unreachable. Q3 and Q4 are measurements and are only as good as the last sample. Keeping the two classes apart is the single structural decision on this page, and every disposition below falls out of it: a gate may be built on intent; a gate built on a measurement has to say what it does when the measurement is missing.
2. (a) The commitment ledger
Section titled “2. (a) The commitment ledger”2.1 The operand, and where every piece of it already lives
Section titled “2.1 The operand, and where every piece of it already lives”committed(N) is the sum, over everything occupying node N, of
reservation × replicas. Both factors are persisted columns on service
(ce ent/schema/service.go:52-61):
service.cpu_request_millicores int NOT NULL DEFAULT 0service.memory_request_bytes int64 NOT NULL DEFAULT 0service.desired_replicas int NOT NULLMEASURED. The request pair is written from the rendered artifact;
ParseComposeServices reads deploy.resources.reservations
(ce modules/app/services_writer.go:81) and UpsertService persists it
(ce modules/app/container_repo.go:187-191). That is the source
docs/decisions/2026-08-04-physical-capacity-gates.md §3 ruled correct, and it
is correct for a reason worth restating: it respects derive only what is
absent. An authored reservation kept verbatim, or one clamped down under its
quota ceiling (X200947QTACLM), is a figure that quota × ratio computes
wrong.
★ And SHC does write reservations now. The brief said it never does, and
that Swarm therefore has nothing to honour. ce modules/app/resource_reservations.go
exists precisely to inject them, and its header carries first-hand
measurements against docker 29.4.1 / compose v5.1.3: reservations.cpus is
ignored by compose entirely and read only by swarm’s scheduler;
reservations.memory becomes HostConfig.MemoryReservation, a soft reclaim
bias that cannot kill (measured: a 64M reservation allowed a 300M allocation,
exit 0). On swarm both keys are hard admission constraints. So the operand
this ledger sums is also the operand swarm’s own scheduler would consult, which
is the property that makes the ledger predictive rather than decorative.
2.2 The attribution problem, and why deployment_node is necessary but not sufficient
Section titled “2.2 The attribution problem, and why deployment_node is necessary but not sufficient”The ledger needs a map from deployment to node. Three candidates exist and only one combination works.
deployment.recorded_node_id: the routing scalar. Measured wrong for occupancy: it names whoever handled the request, and an unpinned swarm install stamps no placement constraint at all. Using it is the defect that let the node-removal gate permit removing a node running live tasks (ce ent/schema/deployment_node.goheader).container.node_id: first-hand per-node attribution (ce ent/schema/container.go:50). Correct while it exists. It stops existing: the leader reaper deletes container rows 120s after the containers go, and an unreachable node’s rows are cleared during the exact outage in which the answer matters most.deployment_node(deployment_id, node_id, anchored, last_observed_at): the set, withanchoredwritten by the same call that writesrecorded_node_idso it is never empty, andlast_observed_atmonotone.
★ The set answers whether, not how much. A 3-replica service spread
across three nodes commits one third of its reservation to each, and
deployment_node has three rows that all say the same thing. Counting the whole
deployment’s reservation once per occupied node triple-counts it. So
deployment_node alone cannot be the ledger’s key either.
2.3 The proposed rule
Section titled “2.3 The proposed rule”PROPOSED.
committed(N) = Σ over services s of reservation(s) × placed(s, N)
placed(s, N) = |{ containers of s observed on N }| ... when any container of s is observed anywhere
= s.desired_replicas if deployment_node(D(s), N).anchored ... otherwise = 0 ... otherwiseIn words: count real replicas where we can see them; fall back to the whole
declared shape on the anchor when we cannot see any. The fallback covers the
three cases the deployment_node header already enumerated (a stopped
deployment, an unreachable node, and the window before a fresh install’s first
recorder pass), and it covers the ordinary compose case, where the anchor and
the runner are the same node and the fallback is exact.
2.4 The deployment_node lesson applies, with the sign of the danger flipped
Section titled “2.4 The deployment_node lesson applies, with the sign of the danger flipped”The recorded rule there is prune on positive evidence, never on absence. It applies here, and the reason it applies is worth writing out because the two gates fail in opposite-looking directions that turn out to be the same direction.
- For the removal gate, a derived-empty set produced a permit; a node with live tasks looked removable.
- For the ledger, a derived-empty sum produces “this node is free”, an admit.
Both are the unsafe direction, so the rule is the same: absence of observation must never reduce a commitment. A stopped deployment still holds its disk and will hold its memory again the instant it starts. An unreachable node’s containers are not gone; they are unobserved.
★ The difference, named rather than glossed: the removal gate’s failure mode is data loss and irreversible; the ledger’s is one deployment too many on a node that then swaps. The ledger can therefore afford a cheaper correctness posture than the removal gate (it does not need its own reconciler), but it must fail in the same direction. Unknown counts as committed, never as free.
2.5 Derived on demand, not materialised, and the argument that decides it
Section titled “2.5 Derived on demand, not materialised, and the argument that decides it”PROPOSED: derive it, from one indexed query, with no new table and no migration.
- Every input is already replicated and already correct.
service,deployment,deployment_nodeandcontainerall live in the same store, and three of the four already CASCADE fromdeployment. An uninstall therefore releases the commitment by deleting the rows the commitment was computed from. There is nothing to credit back, nothing to drift, and no release path to get wrong. That is the single strongest argument and it is worth more than the query cost. - A materialised ledger is
usageagain, on a second axis. MEASURED, the cost of that shape:usageneeded its check and its charge folded into one statement batch (shc modules/quota/enforcement.go,UpdateUsageOnScale) after a measured race in which eight concurrent 500m charges under a 2000m cap all readused = 0and all admitted, and a mutex could not have fixed it, because the racing charges come from different daemons against one shared store. It also needs a reconcile cron and a repair verb. A node ledger would need all three again, for a fact that is derivable. - The read rate does not justify it. The ledger is read at admission and
at rebalance-planning time: tens of reads a minute, not thousands a second.
INHERITED: both estates run 33 deployments (memory
atomic_move_design). The join is over(deployments × services).
★ This does not dodge the “derived-only cannot be a gate” ruling; it honours it one layer down. What could not be derived there was the set. Here the set is an input, not the output; the output is a byte figure derived from a set that is itself durable and never empty. The whole argument rests on the anchor row being written unconditionally. If a path is ever allowed to create a deployment without an anchor row, this design is wrong and the ledger has to be materialised.
2.6 The reader seam
Section titled “2.6 The reader seam”PROPOSED, in ce package app, alongside the existing repo readers:
type NodeCommitment struct { NodeID string CPUMillicores int64 MemoryBytes int64
// Deployments counted, and how each was attributed, so a caller can // say "3 of 11 were counted from the anchor because no container of // theirs was observed" instead of publishing a bare number. Deployments int AnchorFallbackCount int
// UndeclaredServices counts services whose reservation columns are 0. // A 0 is NOT a measurement here (see below), so a consumer that wants // to be a gate can refuse to be one when this is a large fraction. UndeclaredServices int TotalServices int
ObservedAt time.Time}
type NodeCommitmentReader interface { CommitmentFor(ctx context.Context, nodeID string) (NodeCommitment, error) CommitmentForAll(ctx context.Context) ([]NodeCommitment, error)}★ UndeclaredServices exists because a zero here is ambiguous and the
substrate must not pretend otherwise. MEASURED:
service.cpu_request_millicores is Default(0) and NOT NULL
(ce ent/schema/service.go:56), and the schema comment says why: “an existing
row predates them and 0 is the honest reading: nothing declared.” But nothing
declared and declared zero are the same bit pattern, and the whole
hostfacts package exists in this codebase to stop exactly that collapse. In
practice ApplyResourceQuotas writes a reservation on every render, so any
deployment installed or updated since the column landed carries a real figure,
but a fleet is not all one vintage, and a ledger that silently under-counts a
pre-column deployment reports a node as emptier than it is. Counting them and
reporting the count is the cheap honest move. RULING OWED (§9.2) on whether
to backfill or make the column nillable.
2.7 Behaviour under move, uninstall and failure
Section titled “2.7 Behaviour under move, uninstall and failure”| event | ledger behaviour | correct? |
|---|---|---|
| uninstall | inputs CASCADE away with the deployment | yes, by construction |
| move, mid-flight | source anchor persists until the uninstall half; target anchor written at restore ⇒ both counted | ★ yes, and deliberately: during the window both nodes really do hold the bytes, and the target really will need the memory. Double-counting is the conservative answer and it self-clears at the commit point |
| move, reverted after the target anchor was written | target over-counted until the anchor row is pruned | acceptable; the existing prune rule (positive evidence, node reachable) owns it, and the error is in the safe direction |
| node unreachable | observation stops; the anchor floor holds; committed does not fall | yes; this is §2.4 |
| daemon restart | nothing to warm; the ledger is a query | yes, and it is the reason this axis, unlike §3, is readable during boot |
3. (b) Live utilization, readable cross-node
Section titled “3. (b) Live utilization, readable cross-node”PROPOSED: do not build a transport. Promote the one in §0.1, and fix the three things that make it unsafe to gate on.
3.1 Defect one: the freshness clock restarts at the merging node
Section titled “3.1 Defect one: the freshness clock restarts at the merging node”MEASURED. StatsCache.UpdateNode stamps stats.UpdatedAt = time.Now()
unconditionally on write (ce modules/metrics/stats_cache.go:128-133). The
fan-in calls exactly that method for every peer entry
(peer_nodes.go, c.cache.UpdateNode(id, ns)). NodeStats.UpdatedAt has no
json tag, so it marshals under its Go field name and does round-trip; the
peer’s own timestamp arrives and is then discarded.
Consequence: a peer whose local tick source has wedged keeps serving its
last-good NodeStats from its own cache; the merging node re-stamps it fresh
every 2 seconds; NodeAge reports 2s for data that is an hour old. The one
staleness signal in the system reads green for the failure it exists to catch.
★ This is read out of the source, not proven against a running daemon. Marked MEASURED-FROM-SOURCE; a live repro is one wedged tick source away and should be part of whatever lands the fix.
Fix (PROPOSED): carry the peer’s UpdatedAt across the merge
(UpdateNodeAt(id, stats, observedAt)), and leave UpdateNode’s behaviour
alone for the local entry, which genuinely is fresh at write time.
3.2 Defect two: unmeasured exists on one payload and not the other
Section titled “3.2 Defect two: unmeasured exists on one payload and not the other”MEASURED. ce core/hostfacts is the package whose entire purpose is that
“unmeasured” is a value and not a zero, and /internal/health’s
SystemMetricsPayload carries UnmeasuredFacts accordingly
(ce modules/health/internal_router.go:80). /internal/_node_stats’s
NodeStats does not. And SystemTickSource.ReadNode does not go through
hostfacts at all; it has its own readMemInfo / readCPUPercent /
readLoadAvg (system_tick_source.go:386-500), a second set of /proc
readers in the same binary, none of which can express a failed read.
So the fan-in delivers MemoryTotalBytes: 0 for a node that could not measure
memory, and a consumer cannot tell it from a node with no RAM. That is the same
class as the defect the physical-capacity gates decision measured in §4 of its
own page (an unknown collapsing into a zero) reaching a scheduler instead of
an installer.
Fix (PROPOSED): NodeStats.UnmeasuredFacts []string, populated in
ReadNode through core/hostfacts, retiring the duplicate /proc readers.
This is a prerequisite for everything else and is independently useful today:
the HUD publishes those zeroes now.
3.3 Defect three: no durability, which is fine, and one consequence that is not
Section titled “3.3 Defect three: no durability, which is fine, and one consequence that is not”The cache survives nothing. After a daemon restart the node map is empty for
one tick plus one peer round-trip. NodeAge returns 24 hours for a missing
entry (stats_cache.go:159), so a fail-closed consumer correctly refuses
during that window.
PROPOSED: accept it, and do not replicate the hot path. Writing every node’s utilization into rqlite every 2 seconds is a write per node per tick through raft, forever, for a fact that expires in twenty seconds. It is the “put the hot loop in the consensus log” mistake, and it is the same instinct the move journal deliberately refused when it chose a node-local file over a table.
★ One durable thing is worth writing, and it is not the sample. Phase 2,
not required for the first gate: a digest, one row per node per minute,
last-write-wins, holding a 5-minute EWMA of CPU and memory plus free bytes per
path-class and a measured_at. One write per node per minute. It exists
because utilization is a claim about an interval, and the 2-second sample
cannot make one. An instantaneous sample says a node is busy now; it cannot
say a node is busy. A rebalancer that plans on instantaneous samples thrashes;
it moves work onto a node that was idle for the 2 seconds it was asked about.
The EWMA is not a refinement; it is the difference between a scheduler and a
coin flip.
3.4 The staleness budget, and what happens past it
Section titled “3.4 The staleness budget, and what happens past it”PROPOSED: 20 seconds for the utilization axis, stated as a named constant
adjacent to the tick interval, and derived rather than picked: 2s tick × 8
missed rounds, which is the HUD’s existing 15s (shc modules/hud/builder/builder.go:33)
plus one peer round-trip. A sample older than that means the reader has not
heard from that node in ten ticks.
★ What a scheduler does past it: REFUSE TO MOVE. Fail-safe is refuse, and the asymmetry is the whole argument.
- Not moving costs an imbalance that persists to the next planning cycle. It is undone by doing nothing.
- Moving on stale data costs a full stateful cutover (stop, archive, ship,
restore, health gate, data gate), possibly onto the node that is actually the
hot one. INHERITED (memory
atomic_move_design): the revert window closes at the commit point, after which failures roll forward. So a move started on a wrong premise cannot be taken back by regretting it.
The two are not symmetric, so the rule is not symmetric.
★ But stale utilization must never block an install. Admission and rebalancing are different verbs with different costs of refusal. An install refused because the daemon restarted three seconds ago is a product defect; a move refused for the same reason is prudence. The dispositions therefore split by verb, not by axis:
| verb | Q3/Q4 missing or stale | Q2 unreadable | Q1 unknown on an axis |
|---|---|---|---|
| install / admission | proceed; warn | proceed; warn | skip that axis (the existing per-axis rule) |
| rebalance / choose a move target | refuse that node as a target | refuse | refuse that axis; refuse outright if both axes unknown |
| drain (operator-initiated) | proceed, warn | proceed, warn | proceed |
★ Drain is the third case and it is the interesting one. The deployments
have to go somewhere; the source node is being removed. Refusing on stale
data strands them on the node the operator is emptying, which is worse than
placing them imperfectly. So drain keeps today’s disposition (place anyway;
today that means pickTarget’s lowest-row-count) and gains a warning, never a
gate. A capacity substrate that makes shc node drain refuse is a regression
even when its reasoning is impeccable.
4. (c) Per-node free disk
Section titled “4. (c) Per-node free disk”4.1 One function, one definition of free
Section titled “4.1 One function, one definition of free”PROPOSED, step 1. Promote clone.DiskFreeBytes (Bavail, ancestor-walking,
already written and already the right shape) into ce core/ next to
hostfacts, as the single free-space probe, returning (bytes, ok) so an
unreadable filesystem is not a zero. Then:
- delete
backup.diskFreeBytes(a literal copy) and call the promoted one; - collapse
maintDiskUsageProbeanddefaultDiskUsageProbe(a second literal pair) onto it; - point
readDiskStatsandinternal_router’s statfs at it.
★ This is the “reuse rather than duplicate” the brief asked for. The sibling lane already wrote the function; the work is making it the only one.
★ And one fleet-visible hazard has to land with it. Switching
readDiskStats from Bfree to Bavail raises every node’s reported
DiskPercent by roughly the root reserve (about 5% on ext4) and
nodeStateFromMetrics thresholds at > 90 → unhealthy and > 80 → degraded
(ce modules/health/internal_router.go:114-124, system_tick_source.go:154-161). A
node reading 87% today reads about 92% after, and flips to unhealthy on
contact. That is the same shape as the limits-operand problem in the
physical-capacity-gates decision: a correct fix that starts refusing on the day
it lands.
PROPOSED resolution: do not move DiskPercent. Leave it on Bfree for wire
stability and thresholds, and add FreeBytes (Bavail) as a new field.
The percentage stays a health signal; the byte count becomes the budget. They
are different questions and it is fine for them to have different answers, as
long as the field names say which is which.
4.2 The fact, per path-class rather than per path
Section titled “4.2 The fact, per path-class rather than per path”PROPOSED. NodeStats.Filesystems map[string]FilesystemFact, keyed by
role, not by path:
type FilesystemFact struct { Role string // "deployments" | "staging" | "backups" MountPoint string // so a reader can see three roles on one mount TotalBytes int64 FreeBytes int64 // Bavail Measured bool MeasuredAt time.Time}Roles, not paths, because the questions are role-shaped (“can the target hold
the restored tree”, “can the executor hold the archive”) and because the
mount point is what tells a reader whether two roles are competing for the same
bytes. On a stock estate all three resolve under /var/lib/shc and carry one
mount point; the day anyone mounts a separate volume for backups, the single
figure in use today is wrong for two of the three and nothing notices.
Cadence: not 2 seconds. Three statfs calls per node per tick plus the
fan-in is noise for a number that moves slowly. PROPOSED: refresh the
filesystem facts every 30 seconds inside the existing tick source (a counter,
not a second loop), and stamp MeasuredAt per entry so the staleness of this
axis is independent of the CPU/memory axis.
4.3 What it is for, and the one capability that does not exist yet
Section titled “4.3 What it is for, and the one capability that does not exist yet”Use 1: the move’s staging preflight, which is already built for the local
side. preflightStagingSpace (ce modules/clone/preflight_space.go) refuses
X507632CLNSPC when the staging disk cannot hold the estimate. It is the right
gate with the right disposition (unmeasurable skips with a warning, because
this gate protects a disk rather than licensing a destruction, the opposite of
the data gate, and its header says so).
★ The gap: it measures the executor’s staging dir and nothing else.
MEASURED: FilesystemProvider.FreeBytes(path string) takes no node
argument (ce modules/clone/types.go:868), while its sibling
MeasureDeploymentTree(ctx, tenant, stack, environment, node) does
(types.go:963) and the full daemon bundle serves it over the mesh peer client.
So a move can measure a remote tree and cannot measure a remote disk.
A rebalancer choosing node N as a move target must know N can hold the restored deployment. Nothing can ask. This is the concrete capability lane 2 owes the move lane, and once §4.2 lands it is a cache read rather than a new RPC:
FreeBytesOn(ctx context.Context, nodeID, role string) (bytes int64, measured bool, err error)served from the fan-in, with measured=false when that node’s entry is stale
past the §3.4 budget or names the role unmeasured. The move’s preflight then
checks both ends and keeps its skip-with-a-warning disposition on the
target side too.
Use 2: placement. A node whose deployments filesystem sits at 92% is not
a move target. ★ And a refuse on disk is defensible in a way a refuse on CPU
is not: CPU exhaustion is a slowdown, ENOSPC is an outage, and the measured
incident that motivated preflight_space.go was a backup that “filled the
staging filesystem MID-CAPTURE… and the failure took the whole node’s disk down
with it; the daemon, the databases and every co-resident deployment share that
filesystem.” Disk is the one axis where this substrate should be allowed to say
no before any ruling on the others.
5. (d) The admission seam
Section titled “5. (d) The admission seam”5.1 The shape, copied from QuotaEnforcer exactly
Section titled “5.1 The shape, copied from QuotaEnforcer exactly”MEASURED, the pattern to match (ce modules/app/quota_seam.go): an
interface in package app; a package-level var quotaEnforcer QuotaEnforcer;
SetQuotaEnforcer(q) that no-ops on nil; unexported helpers (enforceQuota,
recordQuotaCreate, …) that no-op when unwired; wired daemon-side from
shc daemon/wiring/quota_wire.go (eeWireQuotaEnforcer); fail-OPEN on nil
for the caps, with recordQuotaRender the one deliberate fail-closed member.
PROPOSED, ce modules/app/capacity_seam.go:
type CapacityRequest struct { Runtime string Node string // pinned / host-bound target; "" on unpinned swarm Members []string // swarm pool member node ids; nil on compose
// Σ reservation × replicas over the render, and the largest single // replica, the two operands the three existing gates already take. TotalCPUMillicores int64 TotalMemoryBytes int64 LargestReplicaCPUMillicores int64 LargestReplicaMemoryBytes int64}
type CapacityVerdict struct { Fits bool Reasons []string // one per failing axis, operator-readable Code string // the coded refusal a refusing posture would return Skipped []string // axes NOT judged, and why: "memory: node w1 reports no total"}
type CapacityAdvisor interface { Advise(ctx context.Context, req CapacityRequest) (CapacityVerdict, error)}5.2 Two deliberate departures from QuotaEnforcer, and why
Section titled “5.2 Two deliberate departures from QuotaEnforcer, and why”★ It is Advise, not Enforce, and the name is the design.
QuotaEnforcer.Enforce returns error: allowed, or a coded 403. One bit. A
physical capacity check has three outcomes that matter (fits, does not fit,
could not tell), and collapsing the third into either of the other two is
precisely the defect the gates decision measured: an unknown read as a zero
refused 100% of memory-bearing installs on every estate, single-node ones
included, while blaming the pool for having 0 bytes. A verdict struct makes
“could not tell” expressible across the repo boundary, which is the only
place it can be handled correctly.
★ The advisor returns a verdict; the CALLER picks the posture. That is the
opposite of the quota seam and it is deliberate. Quota posture is not
configurable; a cap is a cap. Capacity posture is: deploy.capacity_admission
∈ off | warn | refuse, default off, option C of the gates decision. Putting
the knob at ce’s call site rather than inside the shc functions means it
resolves through the ordinary 5-level cascade at the one point where tenant,
environment and stack are all in hand, and means the three gates never have to
know a policy exists.
5.3 The call site
Section titled “5.3 The call site”INHERITED from docs/decisions/2026-08-04-physical-capacity-gates.md §2:
the window is after placement is decided and before the first filesystem write,
so a refusal rides the existing pre-deploy refusal contract: nothing created,
the QUEUED intent row deleted rather than stamped FAILED.
MEASURED, re-located at ce 6a3dcc5 (that page’s line numbers were taken at
8ab2081 and have moved):
| line | step |
|---|---|
install_runner.go:1275 | ApplyResourceQuotas: mints quota, writes derived limits + reservations |
:1318 | chargeRenderQuotas: the tenant-cap charge |
:1455 | StampPlacement → renderedPlacement: placement is decided here |
:1507 | SwarmLoaderRefusals: the existing swarm-only preflight |
:1520 | deployDir := DeploymentDir(...): first filesystem write |
The window is (1455, 1520), and :1507 is an existing refusal already
sitting inside it, so the new call has a neighbour with the right disposition to
sit beside.
5.4 The three existing gates: adopt two, re-sign one, add a fourth
Section titled “5.4 The three existing gates: adopt two, re-sign one, add a fourth”PROPOSED.
-
EnforceLargestReplicaFitsNode: adopt verbatim. Its question is “is any node physically big enough for one replica”, and the answer does not depend on what is running: a 16 GiB service does not fit an 8 GiB node whether that node is empty or full. Knowing usage cannot improve it. It is also the only one of the three that catches a failure with no other detection (swarm accepting the service and leaving the taskPendingforever), and its per-node unknown handling (an unmeasured axis counts as satisfied, because a node cannot be excluded unless provably too small) is already right. -
EnforceSwarmPoolCapacity: adopt verbatim, understood as a floor. It is a necessary condition, not a sufficient one: it catches “the whole fleet cannot hold this stack”, which no per-node check catches because the stack could legitimately be spread. Its stricter unknown rule (an axis is comparable only when every member reports it) is correct for a sum and should not be loosened. -
EnforceNodeCapacity: re-sign it. This is the one that must change, and the change is a signature change, not a policy change. Today it compares a request against TOTAL and reads no usage. ★ Its own doc comment is the specification: “Turning them into oversubscription checks means subtracting committed reservations, which needs an operand no signature here accepts; a call-site change, not a change in this file.” So:EnforceNodeCapacity(ctx, node NodeCapacity, committed NodeCommitment,cpuMillicores, memoryBytes int64) errorcomparing against
total − committed. ★ And the unknown handling has to grow a third state, which that comment does not mention. Total known plus committed unknown is not “committed = 0”; that is the unknown-as-zero error one layer out, and it admits everything instead of refusing everything. The per-axis predicates generalise: an axis is judgeable only whenknowsTotal(axis) && knowsCommitted(axis), and an unjudgeable axis lands inCapacityVerdict.Skippedwith its reason. -
The fourth gate, which does not exist: headroom.
total − committed ≥ requestadmits a node to exactly 100% reserved. Every real scheduler keeps a floor for the system itself: the daemon, rqlite, the docker engine, the collector, the OS. PROPOSEDEnforceNodeHeadroom, subtracting a system allowance before any deployment is counted. ★ Not a percentage: 10% of acpx32and 10% of acx53are different amounts of daemon, and the daemon’s footprint does not scale with the box. An absolute floor, resolved through the ordinary cascade asdeploy.node_system_reserve.cpu/.memory. RULING OWED (§9.3) on the numbers; I did not measure the daemon’s own footprint on either estate, and that measurement is a prerequisite to choosing them, not a detail to be filled in later.
5.5 What the gates still cannot do after all of this
Section titled “5.5 What the gates still cannot do after all of this”They judge a request against a node. They do not choose the node. Scoring, spread, anti-affinity, concurrency caps and the rebalancing decision itself belong to another lane. This substrate’s job is to be the thing that lane reads, and to refuse cleanly when it cannot answer.
★ MEASURED, and worth stating because it is the whole reason lane 2 exists:
the only node-scoring code in the product is pickTarget
(shc modules/cluster/drain.go:512): lowest count of running deployment
rows, tie-break by node id, prefer matching runtime, filtered to
availability=active, with the count supplied by
DeploymentLister.ListByNode (drain.go:493-502). No CPU, no memory, no disk,
no quota, no affinity. It is not exported, and it is never consulted at install.
A row count is a proxy for load that is exactly wrong in the case that matters:
one gitlab is not one test-simple.
6. The minimum viable substrate, in build order
Section titled “6. The minimum viable substrate, in build order”Ordered by dependency. Each step is independently landable, independently useful, and independently revertible.
Step 0: the honesty fixes. No dependencies, no new consumers, ship first.
NodeStats.UnmeasuredFacts through core/hostfacts (§3.2), and preserve the
peer’s UpdatedAt across the fan-in merge (§3.1). ★ Ship this before anything
else, because it is the step whose absence makes every later step silently
wrong: a scheduler reading a fabricated zero or a re-stamped stale sample is
worse than one with no data, since it acts confidently.
Step 1: one free-space function, and one new field. Promote
clone.DiskFreeBytes to core; delete the two duplicates; add FreeBytes
(Bavail) to NodeStats without moving DiskPercent (§4.1). Add the
per-role Filesystems map at 30s cadence (§4.2).
Step 2: the commitment reader. app.NodeCommitmentReader, one indexed
query, the anchor-floor rule, the UndeclaredServices count. No new table, no
migration. Read-only; nothing consumes it yet, so it can land and be measured
against both estates before anything depends on it.
Step 3: the seam and the gates. app.CapacityAdvisor + SetCapacityAdvisor
- the ce call site in the (1455, 1520) window at posture
warn, wired shc-side indaemon/wiring/.EnforceNodeCapacityre-signed to take the commitment; gates 2 and 3 adopted unchanged;EnforceNodeHeadroomadded once §9.3 is ruled.
Step 4: the node-addressed free-space read. FreeBytesOn(ctx, node, role)
served from the fan-in, and the move’s staging preflight extended to check the
target as well as the executor (§4.3). This is the step the move lane is
actually waiting on.
Step 5: phase 2, not required for admission: the durable digest. One row
per node per minute: EWMA CPU/memory, free bytes per role, measured_at. This
is what a planner reads across restarts and across time; no gate needs it.
Deliberately not in the minimum: anti-affinity, spread, per-node ratios, a replicated per-tick utilization table, and a scheduler.
7. Rejected alternatives, with their costs
Section titled “7. Rejected alternatives, with their costs”- A materialised
node_commitmenttable, charged at install and released at uninstall. Rejected; §2.5. It isusageon a second axis, andusageneeded a measured race fix, a reconcile cron and a repair verb to stay honest. Cost of rejecting: every read is a join rather than a lookup. Immaterial at 33 deployments; revisit at four figures. - Gossip. Rejected. There is already an mTLS mesh with a working peer
client and a working fan-out; adding an eventually-consistent membership
protocol to distribute a fact that expires in 20 seconds buys nothing the 2s
pull does not, and costs a partition-time divergence mode in a product whose
linearizable substrate is already present. Cost of rejecting: the pull is
O(N²); every node polls every peer every 2s, because
LeaderOnly()isfalseby choice. At 3–10 nodes that is nothing. At 100 nodes it is 10,000 requests per 2 seconds and the fan-in must become leader-only. The flag to do that already exists; the threshold at which to flip it does not, and should be measured before any estate approaches it. - Read utilization back out of the OTel pipeline. Rejected. The right numbers with the right labels are already exported (§0.3), but B9 removed the scrape endpoint deliberately, and making placement depend on an external TSDB makes placement unavailable whenever telemetry is, on a product whose premise is self-hosting without external dependencies. Cost of rejecting: we recompute in-process what is already computed for export.
- Ask every node synchronously at decision time. Rejected. It turns a placement decision into an N-way network fan-out on the install path, with the slowest node setting the latency and an unreachable node forcing a policy choice under time pressure. Cost of rejecting: decisions run on data up to 20 seconds old, which is exactly why the utilization axis is advisory at install and gating only at rebalance (§3.4).
- Wire the three existing gates as-is at
refuse. Rejected; option A of the gates decision, and it stays rejected for the reason measured there plus one this page adds: against TOTAL they answer the wrong question, and a gate that answers the wrong question correctly is worse than no gate, because it gets switched off and takes the right question’s answer with it. - Delete the three gates. Rejected.
EnforceLargestReplicaFitsNodehas no substitute and no other detector. - A node-scoped reservation ratio. Rejected, and not by this page; the
owner ruled it out 2026-08-03 (INHERITED, memory
reservation_ratio_ruling): node is an infrastructure axis, not a config scope, and a per-node ratio papers over a wrongly-sized limit. The per-node concern is real and belongs in this substrate’s headroom knob (§5.4), which is a property of the node, not of the tenant’s derivation.
8. What this substrate cannot do
Section titled “8. What this substrate cannot do”Stated plainly, because each of these will otherwise be discovered by a consumer that assumed otherwise.
- It cannot tell a declared-zero reservation from an undeclared one; §2.6. Mitigated by counting them, not solved.
- It cannot split a stopped multi-node swarm deployment’s reservation across its nodes. The anchor floor covers whether; the per-node split needs the container table, and the container table is reaped 120s after the containers stop. Named, not solved. In practice it does not bite today: INHERITED, both estates are 100% compose-runtime with zero swarm-runtime stacks, and no swarm-runtime install has been verified end to end.
- It has no memory. Every figure is either instantaneous or intent-derived. “Was this node hot yesterday” is unanswerable until step 5, and the first rebalancer built on this will thrash if it is written before step 5 lands.
- ★ It measures SHC-managed load only. A hand-run container, a co-tenant
process, the operator’s own compose file, all invisible to
committedand visible toutilization. That is a good reason the two axes are not interchangeable and a better reason the utilization axis has to exist at all: the ledger knows what SHC promised, and only the measurement knows what the machine is actually doing. - It cannot make swarm honour any of it beyond what swarm already reads.
SHC does write
deploy.resources.reservations(§2.1), and swarm treats both keys as hard admission constraints, so the substrate and swarm’s scheduler would at least be reading the same numbers. But INHERITED (the twodocs/architecture/cluster/pages, “landed understanding, not proposals”): swarm is a second-class runtime, the fabric is the strategic multi-node path, the fabric is a data plane and not a scheduler, and the two are mutually exclusive atshc initwith no in-place flip. This substrate is deliberately runtime-neutral for that reason: everything in §2 and §3 is computed from SHC’s own tables and SHC’s own mesh, and none of it requires swarm to exist. - It does not decide anything. No scoring, no spread, no anti-affinity, no concurrency cap, no never-move annotation, no kill switch. Those are the prerequisites the owner named, and they are a different lane’s list.
9. Rulings owed
Section titled “9. Rulings owed”9.1: the reservation multiplier (§0.4). Blocks: the meaning of every number
this substrate produces. Shipped is reservation = quota × 0.25,
limit = quota × 1.0. The owner’s stated model is
reservation = quota × 1.0, limit = quota × 1.5. The commitment ledger is
correct under either, but a node’s committed sum differs by 4× between them,
which decides whether any capacity gate ever fires and what a headroom number
means. This is not a quota question (the charge is the quota either way); it is
a capacity question, and it should be answered before step 3.
9.2: undeclared reservations (§2.6). service.cpu_request_millicores is
Default(0) NOT NULL, so “declared zero” and “row predates the column” are
indistinguishable. Backfill from the render, make the column nillable, or accept
the ambiguity and gate on the UndeclaredServices fraction. Interim behaviour is
proposed (count and report); the durable answer is owed.
9.3: the system headroom allowance (§5.4). How much CPU and memory a node reserves for the daemon, rqlite, docker and the OS before any deployment is counted. ★ Not answerable from a desk: it needs the daemon’s own footprint measured on both estates first. PROPOSED placeholder 1000m / 2 GiB, explicitly unmeasured.
9.4: the admission posture (carried, unchanged). off | warn | refuse for
deploy.capacity_admission. The gates decision recommends D-then-C-at-warn
and that recommendation is unchanged by this page. What this page adds is that
disk deserves a different answer from CPU and memory (§4.3): ENOSPC is an
outage rather than a slowdown, the incident that motivated the existing
free-space preflight was exactly that, and disk is the one axis where refuse
is defensible on a first roll.
Related
Section titled “Related”docs/decisions/2026-08-04-physical-capacity-gates.md: the three gates, their operand, their unknown-handling fix, and the posture question this page inheritsshc modules/quota/enforcement.go: the three gates and theNodeCapacityunknown predicatesce modules/app/quota_seam.go: the seam shape §5 copiesce modules/app/resource_quota.go,resource_reservations.go: where the operand comes from and what the runtimes do with itce ent/schema/deployment_node.go: the occupancy set and the prune rule §2.4 reasons fromce modules/clone/preflight_space.go: the free-space preflight §4 reuses rather than duplicatesce modules/metrics/collectors/peer_nodes.go,ce subsystems/impl/peer_node_stats_adapter.go: the fan-in §3 promotesce core/hostfacts: the “unmeasured is a value, not a zero” package §3.2 extends to the second payloaddocs/architecture/cluster/swarm-quorum-and-health.md,swarm-contention-map.md: the runtime fork this substrate stays neutral across