The three physical capacity gates: wire, warn, or delete
Superseded by New-line runtime. The text below is the record as of its date.
Status: open owner decision. Cost of delay: cosmetic today (the gates are inert, so nothing is broken by waiting); blocks-work for the swarm preflight lane, which has nothing else to refuse an unschedulable stack with.
Baselines. shc origin/main 6a559b526, ce origin/main 8ab2081.
Every figure below was measured against those two trees, not inferred.
★ UPDATE 2026-08-04: the two blocking prerequisites are DONE. The posture question is unchanged and still open.
Steps (1) and (2) of option D have landed. Nothing is wired; no call site was added; the codes remain unreachable.
- The per-axis unknown guard is fixed (
shc modules/quota/enforcement.go).hasResources()is gone, replaced by per-axisknowsCPU()/knowsMem()predicates that also read a pointer-to-0as unknown. §4’s measured refusals no longer reproduce: the 1-core/1-GiB request on a 48-core pool with memory unknown is admitted by all three gates. The known axis still enforces;modules/quota/enforcement_unknown_axis_test.gocovers known-enforces, unknown-skips and the mixed case per gate, and every assertion was verified red against this page’s baseline tree first.node.memory_bytesexists (ce, migration20260804022217_auto.sql, additive,Int64Nillable, NULL = unknown). It is written by the same three writers ascpu_cores(the genesis upsert atshc init,node_self_register’s boot upsert, and the per-node fact-refresh tick), so an upgraded estate backfills itself within one tick and a resized host re-converges without a daemon restart. The probe (cluster.PhysicalMemoryBytes) returns 0 for unmeasurable, never a guess, and 0 persists as NULL.Still open, and still the owner’s call: the posture (A/B/C/D-then-B/E), the ce
CapacityAdvisorseam, and the call sites in §2. Still true and now recorded in the code: the operand must be the reservation (§3), anddocs/architecture/flows/install.md§10–§12 still documents these gates as things the install path does (§2, last paragraph) and is still wrong.
EnforceNodeCapacity (shc modules/quota/enforcement.go:117),
EnforceSwarmPoolCapacity (:138) and EnforceLargestReplicaFitsNode
(:184) are well-written, unit-tested, and have zero non-test callers in
either repo. Their codes X403010NODCPU, X403011NODMEM,
X403012SWMPOL, X403013SWMREP are unreachable. This page states where
they would be called, what they would refuse, and why the answer is not
yet rather than yes or delete.
The recommendation in one line: do not wire them this release; land the
per-axis unknown-guard fix (zero production risk, no callers), add a durable
node memory column, and only then wire them at warn, with the
reservation as the operand, never the limit.
1. What each gate asserts, and in what units
Section titled “1. What each gate asserts, and in what units”All three compare a request against a node’s TOTAL capacity. None of them subtracts what is already committed; there is no usage read in the file. So they are not oversubscription gates; they are does this even physically fit an empty box gates. The error templates say so in as many words: “node {node} only has {capacity} millicores total”.
| Gate | Signature | Asserts | Nil-safe? |
|---|---|---|---|
EnforceNodeCapacity | (node NodeCapacity, cpuMillicores, memoryBytes int64) | the request fits one named node | yes; was per-field nil-guarded already; now also treats a pointer-to-0 as unknown |
EnforceSwarmPoolCapacity | (members []NodeCapacity, totalCPUMillicores, totalMemoryBytes int64) | the request fits the sum of the pool | was no, now yes; see §4 |
EnforceLargestReplicaFitsNode | (members []NodeCapacity, perReplicaCPUMillicores, perReplicaMemoryBytes int64) | some single node satisfies CPU and memory together | was no, now yes; see §4 |
Units: CPU in millicores (NodeCapacity.CPUMillicores = cpu_cores × 1000), memory in bytes. NodeCapacity’s fields are pointers so
“unknown” reads as nil rather than as a legitimate zero.
Gate 3 exists for the case gate 2 cannot catch: the aggregate pool is large
enough but no single node satisfies both axes at once, so swarm accepts the
service and leaves the task Pending forever.
2. The three call sites
Section titled “2. The three call sites”The install/update render chain is ce modules/app/install_runner.go.
Ordering at ce 8ab2081, and the ordering is the whole answer:
| Line | Step |
|---|---|
| ~1025 | ApplyResourceQuotas: mints per-service quota; writes limits = quota × limit_ratio and reservations = quota × reservation_ratio |
| 1068 | chargeRenderQuotas: the tenant-cap charge (see §3 on its operand) |
| ~1204 | StampPlacement → renderedPlacement PlacementDecision{Node, Pins}; placement is decided here |
| 1256 | SwarmLoaderRefusals: swarm-only preflight refusal |
| ~1268 | deployDir := DeploymentDir(...): first filesystem write |
So the correct window is (1204, 1268), after placement exists, before
anything is created, which is what lets a verdict ride the existing
pre-deploy refusal contract (install_refusal.go: nothing created, the
QUEUED intent row deleted rather than stamped FAILED).
EnforceSwarmPoolCapacityandEnforceLargestReplicaFitsNode: immediately afterStampPlacement, inside the existingif opts.Runtime == string(RuntimeSwarm)branch that already hostsSwarmLoaderRefusalsat:1256. Swarm-only by nature: on compose there is one node and swarm’s scheduler is not making the decision.EnforceNodeCapacity: the same window, on the pinned/host-bound branch.renderedPlacement.Pinsnames, per service, the node(s) the rendered artifact pins to; for a host-bound runtime the target isopts.PlacementNode.
All three need a new seam. ce cannot import shc/modules/quota. The
shape already exists and should be copied, not invented: QuotaEnforcer in
ce modules/app/quota_seam.go, wired daemon-side by
shc daemon/wiring/quota_wire.go (eeWireQuotaEnforcer), fail-open on
nil. A CapacityAdvisor alongside it is the minimal addition.
Nobody reading the docs would know they are unwired, which is likely how
they stayed inert this long. docs/architecture/flows/install.md §10–§12
describes pool capacity, largest-replica-fits and EnforceNodeCapacity as
things the install path does, with line numbers, and names
X403014SWMFRG, CheckSwarmCanPlaceReplicas, EnforceStackCreationQuota,
ExtractComposeResources and PreRenderCompose. All five have zero
occurrences in either Go repo; that page documents the Python original as
if it were the Go path. It should be corrected whichever way this is ruled.
3. The operand: reservations, not limits, and the call site has the wrong one
Section titled “3. The operand: reservations, not limits, and the call site has the wrong one”The figure sitting in scope at the charge is renderQuotaTotals(quotas, replicas), Σ (quota × replicas). At a limit_ratio of 1.0 (the default then; it is 1.5 now)
the quota is the limit, so that figure is the sum of the LIMITS. It
is the correct operand for the accounting charge (the charge is always the
quota; quota_render_charge.go) and the wrong operand for a physical
gate. Four reasons, in ascending order of teeth:
-
Swarm’s scheduler never consults limits.
reservationsis a hard admission constraint;limitsis a post-placement cgroup setting the scheduler ignores. A gate on limits does not predict thePending-forever failure it exists to catch. -
The overcommit is the design, and its size is a declared knob.
limit_ratio ≥ 1(default 1.5) withreservation_ratiodefaulting to 1.0 means the intended overcommit factor is 1.5×. (This said 0.25 and 4× when it was written; ce 358b0f081 raised the reservation default to 1.0 and the limit default is 1.5.) Summing limits measures the deliberately-overcommitted number and reports it as a capacity breach. -
Measured, on
compose01’s real shape (16 cores / 30 GiB), with both axes known:app (catalogue figures) Σ quota quota operand reservation operand greenbone34000m / 32.62 GiB REFUSE ADMIT mailcow8350m / 7.56 GiB ADMIT ADMIT gitlab2250m / 16.25 GiB ADMIT ADMIT huly7000m / 9.99 GiB ADMIT ADMIT And gate 3, per-replica, on the smallest node either estate could use (Hetzner
cpx32, 8 vCPU / ~15.6 GiBMemTotal):gitlab’s largest single service (16 GiB) is REFUSE on the quota operand and ADMIT on the reservation operand. Both are installs that succeed today. -
The estates are already red on the limit sum before anyone installs anything.
docs/reference/quota-model-state.md:325measures 20.68 GiB of memory limits on a 30 GiB node and 18.20 cores on 16. A gate on limits starts refusing on contact.
The reservation is available, from the right place.
ParseComposeServices already returns CPURequestMillicores and
MemoryRequestBytes, parsed from deploy.resources.reservations
(ce modules/app/services_writer.go:118-121), and
renderReplicaCounts already calls it. Reading the reservation off the
rendered artifact (rather than recomputing quota × ratio) is what keeps
the gate and the service row from disagreeing, and it is the only source
that respects derive only what is absent: an authored reservation kept
verbatim, or one clamped under its ceiling, is a figure quota × ratio would
get wrong.
Note ServiceQuota deliberately does not carry the derived reservation
(resource_quota.go:206-213); it did, and it was deleted as a
written-by-one-read-by-none field. Do not re-add it; read the document.
4. The blocking precondition: the memory axis has no durable source, and two gates refuse everything without it
Section titled “4. The blocking precondition: the memory axis has no durable source, and two gates refuse everything without it”FIXED 2026-08-04. Both halves of this section are closed; see the update box at the top. The measurements below are retained because they are the evidence for the ruling, and because they are the shape any future capacity consumer must not re-create.
There was no node memory column. ce’s node table carried cpu_cores
(migrations/20260730032524_node_cpu_cores.sql, Nillable, NULL = unknown,
never zero) and nothing for memory. The only memory-total source in the
product was shc modules/metrics’ StatsCache →
NodeSnapshot.MemoryTotalBytes, an in-memory cache with a 15-second
staleness threshold (shc modules/hud/builder/builder.go:33) that a
consumer on another node cannot read at all. node.memory_bytes now exists
alongside cpu_cores with the same NULL-is-unknown contract and the same
three writers.
And the unknown-resources guard could not survive a half-known node.
hasResources() was cpu > 0 || mem > 0 (:35-37), and the skip was
evaluated pool-wide, across both axes at once (:82-84, :121-123).
CPU known + memory unknown ⇒ hasAnyResources == true ⇒ the memory
comparison runs against poolMem == 0 ⇒ every memory-bearing request is
refused. Measured against a 3-node pool of 16 cores each with memory
unknown (i.e. exactly what the node table can supply), asking for 1
core and 1 GiB:
GATE1 EnforceNodeCapacity: admitted (per-field nil guard holds)GATE2 EnforceSwarmPoolCapacity refused: [X403012SWMPOL] swarm pool capacity exceeded, requires 1000 millicores and 1073741824 bytes but pool has 48000 millicores and 0 bytes across 3 nodesGATE3 EnforceLargestReplicaFitsNode refused: [X403013SWMREP] no single swarm node can host one replica, requires 1000 millicores and 1073741824 bytes but largest node has 16000 millicores and 0 bytes across 3 nodesSo the naive-wiring failure was worse than oversubscription: not a policy error that refuses a few large stacks, but a refusal of 100% of memory-bearing installs on every estate, including single-node ones, with a message blaming the pool for having 0 bytes.
The fix, landed: the guard is per-axis. knowsCPU() / knowsMem()
replace hasResources(), and each axis is judged only when it has a usable
total (a nil pointer and a pointer-to-0 both read as unknown). Two
details fell out of doing it honestly rather than mechanically:
- Gate 2’s pool sum needs EVERY member to report the axis, not just one. A sum over a set containing an unknown is unknown; adding up only the members that answered under-states the pool and would refuse a stack that fits, the same unknown-as-zero error, localised to one member. One unmeasured node abstains the axis for the whole pool.
- Gate 3’s per-node test must treat an unknown axis as SATISFIED, because the question is “does some node fit” and a node cannot be excluded unless it is provably too small. A refusal now means every member failed on an axis that was actually measured.
Because there are zero production callers, that fix carried no production risk, which is why it landed on its own ahead of any posture ruling.
5. Options
Section titled “5. Options”| Posture | Verdict | |
|---|---|---|
| A | Wire and refuse | No. Refuses installs that work today even with the right operand on a right source. (It no longer refuses everything; that was the guard defect, now fixed, but the reason to say no was never only that.) |
| B | Wire advisory warn-only, coded | Viable. X200944 is the one free code below the quota warn band (X200945–X200948; X200949/X200950 are taken), emitted via corelog.Warn exactly as X200946QTASRC and X200948QTALRT already are. Warn-vs-refuse is still the smallest part of the work, but the remaining part is now just the seam; the source and the guard fix are done. |
| C | Wire behind an off-by-default scoped knob: deploy.capacity_admission ∈ off|warn|refuse, default off, through the same 5-level cascade the other deploy.* keys use | Viable, and lets one estate opt into refuse after measuring. |
| D | Do not wire; fix preconditions in order: node.memory_bytes, NULL = unknown, written at enroll/self-register and re-converged by the existing per-node fact-refresh, mirroring cpu_cores | Recommended, and steps 1–2 have landed, so this now reads “do (3), then rule on (4)”. |
| E | Delete all three | No; see below. |
6. Recommendation
Section titled “6. Recommendation”D, then C at warn. Specifically:
- Not this release. (As written 2026-08-04 before the fixes: the memory
axis had no durable source, and the two gates that would consume it refused
everything when it was absent; §4.) Both are now closed, so the
break-a-working-fleet objection is gone; what remains is the operand and
the posture. A physical gate that is wrong once gets switched off
permanently rather than fixed, which is still the argument for
warn. Land the per-axis guard fix first and alone.Done 2026-08-04, together with the durable memory column. Zero production callers, measured behaviour, no posture implications; the two landmines for whoever wires next are cleared. This was the only part of this page that was pure engineering; everything left is a decision.- Keep them; do not delete. On swarm they are the only thing between an
operator and a task
Pendingforever, and the swarm path already needs more preflight, not less; 37 of 73 catalogue apps cannotdocker stack deployat all (M27,docs/architecture/cluster/swarm-contention-map.md). M33 on that page offers exactly this fork: “give them a call site and a node-capacity source, or delete them.” The source is the missing half, and it is buildable. - When wired, default
warn, and neverrefuseon a first roll. Not caution in general; the specific stack of derivations: the operand is read off a document SHC itself rewrote three transforms earlier, compared against a capacity figure from a 15-second cache, on a fleet whose declared overcommit was 4× then (1.5× at today’s defaults). - The reservation is the operand. If the owner rules
refuse, this is the load-bearing detail, because the limit sum is red on both estates before anything is installed (§3.4).
7. Two questions this surfaces and does not answer
Section titled “7. Two questions this surfaces and does not answer”- Does a tenant lowering
reservation_ratiobuy themselves more physical admission? The 2026-08-03 ruling makes the ratio tenant-overridable (bounded by quota). Under any reservation-operand gate, a tenant who lowers their ratio shrinks the number the gate measures; they can admit more physical load by declaring they need less of it. Whether that is intended is a policy call. - Should the compose runtime get a gate at all? These are physical-fit checks against total capacity, and compose has one node whose capacity the daemon is running on. Gate 1 is the only one that makes sense there, and what it would catch is narrow: a single service whose reservation exceeds the whole box.
Related
Section titled “Related”shc modules/quota/enforcement.go: the three functionsshc docs/reference/quota-model-state.md§4: the inert-gates statement and the measured 20.68 GiB / 18.20 cores commitmentshc docs/architecture/cluster/swarm-contention-map.mdM33: wire-or-deletece modules/app/quota_seam.go: the seam shape to copyce modules/app/install_runner.go:1204-1268: the call-site windowshc docs/decisions/2026-08-04-owner-docket.md: the rest of tonight’s docket