The placement policy and the scorer: what chooses a node, at install and at rebalance
Status: open design. The first-placement scorer is built, as the decision named below records. Rebalance scoring and signals S6 and S7 are not.
★ SUPERSEDED IN PART: read
Section titled “★ SUPERSEDED IN PART: read docs/decisions/2026-08-11-rebalancer-v1-foundations.md first”docs/decisions/2026-08-11-rebalancer-v1-foundations.mdfirstThis page is the DESIGN DRAFT. The lane that built from it landed on 2026-08-11 and measured several of its claims differently. Where this page and that decision disagree, the decision is what was measured and this page is kept for its reasoning, not for its facts.
The corrections that matter here:
- §1.4’s correction is itself now stale in one direction: the retired keys are retired, but the SHIPPED default multipliers were 0.25 / 1.0, not the owner’s 1.0 / 1.5. Both are now the owner’s numbers.
- §2.4’s signal table is built for S1, S2, S3, S4 and S5. S6 (free disk) and S7 (live utilization) are not, and S7 is still recommended against for the reason §8.3 gives.
- §2.5’s weights shipped verbatim (memory 40 / cpu 20 / count 20 / tenant 10 / app 10, bin-pack as the same function with the first three negated), and so did §2.6’s five determinism properties and §2.7’s abstain-and-renormalise rule.
- ★ §2.7 has a hole this page did not name, found by mutation-testing the implementation. An abstaining candidate carries a placeholder zero into the value slice, and letting that zero into the min/max computation makes ONE unmeasured node rescale every measured node toward the top of the range: a nearly-full node at 1 GiB free and an empty one at 31 GiB go from 1000 apart to 32 apart. Abstention must exclude the candidate from the SCALE, not merely from the weighted sum.
- §4.2’s warning about the wrong door was exactly right and is now a mutation-proven guard.
- §4.4 (
shc placement explain) is NOT built. The data is carried out of the scorer (every candidate’s score and every abstained term with its reason), but nothing renders it.- §4.5’s kill switch shipped as
deploy.placement_scoring, defaultonrather thanoff, withoffmeaning do-not-score exactly as specified.Everything else on this page stands, including the parts the implementation did not reach; they are named in §8 of the decision.
As written: PROPOSED. Nothing in this document is built, and nothing in it should be built before the four rulings in §9 are made. Three of those rulings are cheap; one (§9.1, bin-pack vs spread) is the policy question the owner has to answer himself because it trades money against blast radius and only he prices both.
Baselines. shc docs/rebalancer-policy off main c0b009ba5; ce main
6a3dcc5. Every figure below was measured against those two trees, on this
box, with no estate contact. Claims I did not measure myself are marked
(inherited) and name their source.
The recommendation in one line: build the scorer as a pure function over durable rows, ship it at install time only, default it to spread, and do not connect it to anything that moves data until it has placed a few hundred installs and an operator can read why it chose each one.
0. Why this is the cheap half
Section titled “0. Why this is the cheap half”The rebalancer is the expensive, dangerous half of the control plane: it needs the atomic move, the standby contract, the concurrency cap, the kill switch. The scorer is none of that. It is a function from (candidate nodes, deployment being placed, current commitments) to a node id. It moves nothing, destroys nothing, and it is exercised on every install rather than on the rare rebalance.
That asymmetry is the whole argument of §c: the scorer should ship, and be proven, before a rebalancer exists to consume it. A rebalancer that inherits an unproven scorer will move a deployment somewhere wrong and the operator will never trust either component again.
1. Ground truth, and four corrections to the brief
Section titled “1. Ground truth, and four corrections to the brief”1.1 Confirmed
Section titled “1.1 Confirmed”| Claim | Verified at | Verdict |
|---|---|---|
pickTarget is the only node-scoring code in the product | shc modules/cluster/drain.go:508-533 | confirmed: lowest Count, tie-break by ID, prefers runtime match, never called outside drain |
| Install places on “whoever handled the request” | ce modules/stack/service.go:1027-1043: a fresh install synthesises a descriptor with an empty NodeID | confirmed |
PlacementConstraint / NodeSatisfies are a drain-time FILTER only | ce modules/cluster/placement.go; consumed at shc modules/cluster/drain.go:456-464 | confirmed |
| Three physical capacity gates have zero non-test callers | shc modules/quota/enforcement.go; docs/decisions/2026-08-04-physical-capacity-gates.md | confirmed |
| No node labels anywhere durable | ce ent/schema/node.go has no labels field; labels come only from docker node inspect (shc daemon/wiring/cluster_drain_constraints.go:82-100) | confirmed, and worse than stated; see §3.2 |
| No per-node free-disk fact | get_health returns DiskFree (ce modules/metrics/rpc_utility.go:183) but has zero non-test callers and no durable column | confirmed |
Zero of the catalogue declares x-shc-volumes or deploy.placement | 104 app.yaml files under shc/apps; grep -rl returns 0 for both | confirmed (the brief says 99 apps; I count 104 at this commit) |
1.2 ★ Correction 1: SHC does write deploy.resources.reservations at ce HEAD
Section titled “1.2 ★ Correction 1: SHC does write deploy.resources.reservations at ce HEAD”The brief says “SHC never writes deploy.resources.reservations, so Swarm’s
scheduler has nothing to honour.” That was true; it is not true at
6a3dcc5. ApplyResourceQuotas injects both keys into the rendered artifact
(ce modules/app/resource_quota.go:455-460), effectiveCPUReservation /
effectiveMemoryReservation are the single writers (:488-540), and
ParseComposeServices reads them straight back out
(ce modules/app/services_writer.go:117,121).
This is the single most load-bearing correction in this document, because it means the operand a real scorer needs is already durable:
service.cpu_request_millicores ce ent/schema/service.go:56service.memory_request_bytes ce ent/schema/service.go:60service.desired_replicas ce ent/schema/service.go:52per service, per deployment, in the replicated table, written at every install and update. The committed-reservation-per-node sum the brief calls missing is a two-join query over rows that already exist. Nothing sums them (that part of the brief is right), but the summands are there and no new probe, no new column, and no new tick is needed to compute the figure.
Not verified: whether the estates are running a daemon new enough to have written non-zero request columns. That needs estate contact, which this lane does not have. Treat “the columns are populated on the estates” as unproven and make the scorer tolerate zeros (§2.7).
1.3 ★ Correction 2: cross-node install dispatch already exists
Section titled “1.3 ★ Correction 2: cross-node install dispatch already exists”The brief implies install-time placement needs new machinery. It does not.
makeRoutingStackInstaller (ce daemon/services_stack_routing.go:100-113)
already POSTs the whole install request to /internal/rpc/stack.install on
target.NodeID whenever that isn’t the local node. This is the path
shc install --node other takes today, and its header comment records the bug
it was written to fix.
So the entire execution path for “install this somewhere else” is built and
in production use. A scorer’s only job is to supply the value that --node
supplies. That is the strongest single argument for shipping install-time
placement first, and I did not expect it going in.
1.4 ★ Correction 3: the quota model in the brief names retired keys
Section titled “1.4 ★ Correction 3: the quota model in the brief names retired keys”The brief states reservation = quota × reservation_multiplier (1.0) and
limit = quota × limit_multiplier (1.5). That is the owner’s stated
architecture (inherited, memory quota_derivation_model) and its
shape is intact at HEAD: quota is the input, both figures are derived, both
resolve through the ordinary cascade, there is no node scope. The key names
and defaults are not:
limit = quota × deploy.limit_ratio default 1.0 ce modules/app/resource_quota.go:174reservation = quota × deploy.reservation_ratio default 0.25 ce modules/app/resource_reservations.go:102quota.limit_multiplier and quota.reservation_multiplier were removed as a
declared breaking change (shc CHANGELOG.md:8685) on the grounds that they
“never did anything.” website/src/content/docs/architecture/modules/quota.md:90-92
still documents the removed pair and is stale; worth a one-line fix by
whoever touches that page next, but out of scope here.
Why this matters to a scorer: the default overcommit factor is 4×, not 1.5×. A scorer that thinks reservations are 100% of quota will pack four times too conservatively.
1.5 ★ Correction 4: pinned_node_id is set on every compose deployment
Section titled “1.5 ★ Correction 4: pinned_node_id is set on every compose deployment”PlacementIntent returns the resolved node for any host-bound runtime
(ce modules/app/placement_intent.go:60-68), and the schema comment is
explicit that this is intended: “A host-bound runtime has exactly one place its
bytes can be, so the node resolved at install IS the placement and lands here”
(ce ent/schema/deployment.go:102-105).
Both estates are 100% compose (inherited, brief + swarm-contention-map.md).
Therefore pinned_node_id is non-NULL for every deployment on both estates,
and a rebalancer that reads it as “the operator chose this node, do not move it”
would refuse to move 100% of the fleet.
The distinction does exist in flight: StackInstallRequest.StatedNodeID
(ce modules/stack/types.go:212) captures what the request named before the
back-fill erases it, but it is never persisted. At rest, the schema
cannot answer “did a human choose this node?” for a compose deployment. §3.4
and §10 treat that as a prerequisite, not a detail.
2. (a) The scoring function
Section titled “2. (a) The scoring function”2.1 What exists
Section titled “2.1 What exists”// shc modules/cluster/drain.go:512func pickTarget(candidates []*candidateNode, runtime string) *candidateNodeSort by Count ascending, tie-break by ID ascending, then return the first
candidate whose Runtime matches, else the first overall. Count is the
number of rows from DeploymentLister.ListByNode whose status is running,
starting, or empty (drain.go:470-506, isRunningStatus at :548). The
caller bumps picked.Count++ after each successful migration “so the next pick
spreads” (drain.go:255).
Three properties are worth keeping and one is a bug in waiting:
- Keep: deterministic tie-break by ID.
- Keep: the running tally mutates within a batch, so a plan of N placements spreads rather than stacking N deployments on the same node.
- Keep: it is a pure function over a slice, trivially unit-testable.
- Fix:
isRunningStatus("")returns true, so a row with an unset status counts as running. That is the right conservative choice for a drain (better to over-count than to leave something behind) and the wrong one for a scorer (it inflates a node’s apparent load). A scorer needs its own predicate.
2.2 What count-only scoring gets wrong: measured
Section titled “2.2 What count-only scoring gets wrong: measured”The catalogue’s authored memory limits, summed per app over services.*.deploy.resources.limits.memory:
| Σ authored memory limits | |
|---|---|
security/greenbone | 32.62 GiB |
devops/gitlab | 16.25 GiB |
ai/dify | 15.56 GiB |
standalone/ollama | 12.00 GiB |
| … | … |
email/mailbox-reconciler | 0.125 GiB |
94 of 104 apps declare a memory limit. Largest / smallest = 261×. Median 2.06
GiB, mean 3.33 GiB. The gitlab figure reproduces
docs/decisions/2026-08-04-physical-capacity-gates.md §3.3 exactly (16.25 GiB),
which is the cross-check that my parse is reading the same thing the earlier
measurement read.
So the brief’s framing (“one deployment is gitlab and another is a 40MB link
shortener”) is real and its magnitude is two and a half orders of magnitude.
tools/kutt, the actual link shortener in this catalogue, declares 0.125 GiB.
But the honest counter-argument, which I ran into by testing it: at large N, count-only scoring is fine. Placing all 82 non-fixture apps across 3 nodes in alphabetical order, least-count produced a 1.16× memory imbalance and least-committed-memory produced 1.015×. The law of large numbers does most of the work.
The defect lives at small N, which is the size SHC actually runs. 3000 random draws, 3 nodes, reservation operand (limits × 0.25), imbalance measured as busiest-node-commitment ÷ quietest:
| deployments | picker | median | p95 | worst |
|---|---|---|---|---|
| 9 | least-COUNT | 2.80× | 9.14× | n/a |
| 9 | least-RESERVATION | 1.63× | 4.00× | n/a |
| 15 | least-COUNT | 2.28× | 5.39× | n/a |
| 15 | least-RESERVATION | 1.33× | 2.58× | n/a |
At the fleet size both estates run, count-only scoring leaves the busiest node carrying roughly 2.8× the committed memory of the quietest, and one draw in twenty leaves it carrying 9×. Weight-awareness roughly halves the median and more than halves the tail.
And a finding that cuts against building an admission gate: with the
reservation operand and a cpx32-class node (15.6 GiB, inherited from the
capacity-gates doc), the busiest node exceeded physical memory in 0.0% of
9-deployment draws and 0.3% of 15-deployment draws. With the limit operand
it exceeded in 49% and 75% respectively, which is exactly the §3.4 argument of
the capacity-gates decision, reproduced independently here.
So: weight-aware scoring is worth building for balance and burst headroom, not
for admission. At current fleet sizes a physical-fit gate would essentially
never fire. That is an argument for scoring and against wiring the three gates
at refuse; it does not change the recommendation in that decision doc, it
supports it.
2.3 The replacement: filter, score, commit
Section titled “2.3 The replacement: filter, score, commit”Three phases, in this order, and the order is the design:
1. FILTER hard constraints. A node that fails any is not a candidate. Produces a candidate set, possibly empty.2. SCORE every surviving candidate gets an integer score. Highest wins.3. COMMIT the winner's tally is mutated in the caller's in-memory ledger before the next placement in the same batch is scored.Phase 3 is pickTarget’s picked.Count++ generalised: after a placement, the
winner’s committed CPU and memory rise by this deployment’s reservation and its
deployment count rises by one. Without it, a batch of N placements all see the
same starting state and all pick the same node. This is the property the
drain picker already has and it must not be lost in the rewrite.
An empty candidate set after phase 1 is not a scoring failure and must not
fall back to “score everything anyway.” It is a refusal with a message naming
which constraint eliminated the last candidate, the shape drain.go:232-238
already uses to distinguish “no active peer” from “no node satisfies the
constraints.”
2.4 The signals
Section titled “2.4 The signals”Every signal below reads a durable, replicated row. Nothing reads a probe,
a cache, or a live container. That is deliberate: a scorer that depends on
StatsCache (per-process, 15s, empty before the first tick, inherited,
capacity-gates §4) is a scorer whose answer depends on which daemon you asked
and how recently it booted.
| # | Signal | Source | Available today? |
|---|---|---|---|
| S1 | Free memory reservation = node.memory_bytes − Σ committed | ce ent/schema/node.go memory_bytes; service.memory_request_bytes × desired_replicas joined through deployment.recorded_node_id | yes, no new column |
| S2 | Free CPU reservation = node.cpu_cores × 1000 − Σ committed | same, cpu_request_millicores | yes |
| S3 | Deployment count | deployment_node occupancy set, anchored = true | yes, and this is a better count than drain’s, see below |
| S4 | Tenant spread: how many of this tenant’s deployments the node already holds | join deployment.tenant_name | yes |
| S5 | App spread: how many instances of this app the node already holds | join deployment.app_name | yes |
| S6 | Free disk | ✗ nothing durable | no: §10 P3 |
| S7 | Live utilization | ✗ get_health exists, uncalled, per-node, not stored | no, and I recommend against it, §8.3 |
On S3 and why it should read deployment_node, not ListByNode. The
occupancy set is one row per (deployment, node) whose bytes or tasks are on that
node, with anchored and last_observed_at as two independent facts
(ce ent/schema/deployment_node.go). Its own header records that
recorded_node_id was unsound in both directions for exactly this class of
question. The drain picker’s count predates it. A scorer should read the set
and count anchored rows; that is “how much has SHC put on this box,” which
is the question.
On S4/S5 and why they are scores, not constraints. “Don’t put both replicas of the same app on one node” is a preference at three nodes and a guarantee nobody can honour at one node. Expressing it as a score degrades gracefully: with room, it spreads; without room, it places anyway and the operator can see the penalty in the explain output. Expressing it as a hard constraint means a single-node estate cannot install a second instance of anything. §3.3 covers the case where the operator genuinely wants the hard version.
2.5 The weights
Section titled “2.5 The weights”Scores are integers in per-mille, not floats. Float comparison makes tie
detection architecture-dependent and makes the audit trail unreproducible;
integers make score(a) == score(b) exact and the tie-break deterministic.
score(node) = w_mem × headroom_mem_permille(node) + w_cpu × headroom_cpu_permille(node) + w_cnt × count_permille(node) + w_tnt × tenant_spread_permille(node) + w_app × app_spread_permille(node)where each term is normalised to [0, 1000] across the candidate set, so no
term can dominate merely by having larger units. Proposed defaults:
| weight | spread (default) | bin-pack | rationale |
|---|---|---|---|
w_mem | 40 | −40 | memory is the axis that actually kills a box |
w_cpu | 20 | −20 | CPU degrades, memory OOMs |
w_cnt | 20 | −20 | keeps behaviour recognisable to anyone who has read pickTarget |
w_tnt | 10 | 10 | tenant blast radius is a real product concern; not negated under bin-pack |
w_app | 10 | 10 | ditto |
Bin-pack is the same function with the first three weights negated; it prefers the node with the least headroom that still satisfies the filters. It is not a second code path, and that is the point: one scorer, one explain output, one test surface.
These weights are a starting point and I have not tuned them. The honest statement is that any weighting in this family beats count-only at the estate sizes measured in §2.2, and that fine-tuning them without production placement data would be fitting noise. The weights should be constants first and become config only if an operator produces a case they get wrong (§8.5).
2.6 Determinism
Section titled “2.6 Determinism”The scorer must be reproducible: same inputs, same output, on any node, on any architecture, in any Go version.
- Integer arithmetic only. No floats in the score. Ratios computed as
(numerator * 1000) / denominatorwith integer division. - Sort on
(score DESC, node_id ASC)withsort.SliceStable, mirroringpickTarget’s existing tie-break so the behaviour is familiar. - Candidate order is normalised before scoring: sort by node id up front, so a registry that returns rows in a different order cannot change the answer even transiently.
- The batch tally is part of the input. Placing a plan of N is
fold(scorer, initial_tally, deployments), and the deployments must be iterated in a stated order (by deployment id) so a replan produces the same plan. - No clock, no randomness, no map iteration in the scoring path.
Property 4 gives the audit trail for free: replaying the fold from the recorded
initial tally reproduces every decision, which is what makes
shc placement explain (§4.4) honest rather than a re-derivation that might
disagree with what actually happened.
2.7 Unknowns
Section titled “2.7 Unknowns”shc modules/quota/enforcement.go already states the rule this file must obey,
and it was learned the hard way: “an unknown must never collapse into a zero.
A zero is a measurement; an unknown is the absence of one. A gate that confuses
them does not degrade to unchecked, it degrades to refuse-everything.”
For a scorer the failure mode is inverted but just as bad. A node whose
memory_bytes is NULL, treated as zero, scores as having no headroom and
never receives a deployment; it silently drops out of the fleet. Treated
as infinite, it scores best on every axis and receives everything.
The rule:
- A node with an unknown axis abstains that axis: its term is omitted and the remaining weights are renormalised over the terms it can answer.
- A node with no answerable axis falls back to
w_cntalone, which is exactly today’spickTarget, i.e. the degraded behaviour is the current behaviour, not a new one. - Σ committed of zero is a legitimate measurement (an empty node) and must
not be conflated with “the request columns were never written.” Distinguish
them by asking whether the node holds any
deployment_noderows at all: zero commitments across zero deployments is empty; zero commitments across nine deployments is unmeasured, and the memory axis abstains.
That last bullet is the one that protects against §1.2’s unproven half: an estate whose daemon predates the reservation writer degrades to count-only scoring rather than to a wrong answer.
3. (b) Constraints, not just scores
Section titled “3. (b) Constraints, not just scores”3.1 Extend PlacementConstraint, do not invent a second vocabulary
Section titled “3.1 Extend PlacementConstraint, do not invent a second vocabulary”ce modules/cluster/placement.go already parses docker’s constraint syntax into
typed values, matches them against a NodeAttrs, skips unrecognised keys rather
than rejecting, and exposes HasHardNodePin. It is 130 lines, it has a test
file, and it is already the filter in front of the drain picker.
Use it verbatim as phase 1 of the scorer. Concretely: filterCandidates
moves out of drain.go into a shared placement package, candidateNode grows
the two commitment fields, and drain calls the shared scorer with the
bin-pack-off weights. Drain’s behaviour is preserved exactly if w_cnt is the
only non-zero weight, which is the migration test to write first.
Three additions to the vocabulary, all in docker’s own dialect so an operator who knows swarm needs no new syntax:
| addition | syntax | why |
|---|---|---|
| operator constraints on a deployment | shc install foo --constraint 'node.labels.disk==ssd' | today constraints exist only on the rendered artifact, so a compose deployment cannot express one at all |
| soft preference | --prefer 'node.labels.zone==fsn1' | scored, not filtered, worth a large positive term, never a refusal |
| anti-affinity | see §3.3 | not expressible in this grammar at all |
3.2 ★ The label gap is worse than “no labels column”
Section titled “3.2 ★ The label gap is worse than “no labels column””daemonNodeAttrs populates Labels from docker node inspect
(shc daemon/wiring/cluster_drain_constraints.go:82-100), which requires a
swarm_node_id. And daemonDeploymentConstraints returns nil immediately for
any non-swarm deployment (:41-43).
On a 100%-compose estate, both halves of the constraint pathway return nothing. The parse and match code is live and unit-tested; the data is structurally unavailable. So the existing constraint machinery is not “wired but unused”; on the estates that exist, it is wired to a source that cannot answer.
This is a prerequisite, not a caveat: node labels need a durable home
(node.labels as a JSON column, or, better, given the config module already
exists, a node-keyed config row) before any label-bearing constraint means
anything outside swarm. Until then, --constraint node.hostname==X and
node.id==Y work (those come off the registry row) and node.labels.* silently
matches nothing.
★ And the silence is the danger. SatisfiesConstraint returns true for an
unrecognised key by design: “skip keys we can’t evaluate rather than reject the
node.” But node.labels.gpu==true is a recognised key; it resolves to the
empty string for a node with no labels (placement.go:86-89) and therefore
fails the == test. So on a compose estate a GPU constraint eliminates
every candidate and the operator gets “no node satisfies the placement
constraints” with no hint that the label source was never populated. The scorer
must detect “this constraint references a label and no node in the fleet has any
labels at all” and say that instead.
3.3 Anti-affinity is not expressible in this grammar, and pretending otherwise is the trap
Section titled “3.3 Anti-affinity is not expressible in this grammar, and pretending otherwise is the trap”NodeSatisfies(constraints, node) evaluates one node against literal values.
Anti-affinity (“these two deployments must not share a node”) is a relation
between a candidate and the rest of the placement, which no (key, op, value)
triple can express. Bolting it in by inventing node.deployments!=foo would
create a constraint whose truth depends on state the matcher does not receive,
and the matcher’s purity is why it is testable.
Two honest options:
Option A (recommended): derive constraints per placement. Before phase 1,
the caller resolves the deployment’s anti-affinity group into a concrete
exclusion list against the current tally, and emits ordinary
node.id!=<id> constraints. NodeSatisfies stays pure; the relation is
resolved by the caller that already holds the tally. Cost: the derivation must
re-run for every placement in a batch (cheap: it is a map lookup), and the
audit trail must record the derived constraints, not just the authored ones,
or an operator reading the explain output sees an exclusion with no stated
cause.
Option B: a second filter phase. filterByRelations(candidates, tally, deployment) runs after NodeSatisfies. More direct, but it splits “why was
this node eliminated” across two mechanisms with two message formats, and the
first thing an operator asks is exactly that question.
Option A. It costs one derivation step and keeps one refusal vocabulary.
Where the group comes from: a deployment-scoped config key
placement.anti_affinity_group (a string; deployments sharing a value repel).
Soft by default (a large negative score term) with placement.anti_affinity: required promoting it to a hard exclusion. Soft by default is not timidity:
at one node, hard anti-affinity makes the second member uninstallable, and one
node is a supported SHC topology.
3.4 never-move: copy deployment.protected exactly
Section titled “3.4 never-move: copy deployment.protected exactly”The owner named a per-deployment never-move annotation. The shape to copy is
already in the tree and its header is a written record of the two ways this goes
wrong (ce modules/app/protected.go):
- a deployment-scoped config row through the same 5-level cascade: no ent migration, and deployment-scoped config is state, so it rides backup/restore for free;
- mirrored onto the manifest sidecar, because the config resolver is nil in
the
shcCLI process and a guard that only exists daemon-side is a guard the daemonless path walks straight through; - fail closed: unreadable is UNKNOWN and UNKNOWN reads as protected;
- the write reports which halves landed, because “armed” that armed nothing readable is the specific lie that file exists to stop telling.
So: deployment.never_move, bool, same cascade, same mirror, same fail-closed
read, same two-store write report. Fail-closed for never-move means an
unreadable flag reads as never_move = true; the rebalancer declines to move
something it cannot ask about, which costs availability-of-optimisation and
never costs data.
3.5 ★ protect already refuses move, and the rebalancer inherits that
Section titled “3.5 ★ protect already refuses move, and the rebalancer inherits that”ce modules/clone/protection.go refuses shc move outright on a protected
deployment, up front, before any phase runs, because the protection flag cannot
reliably follow a move across a rename/tenant/environment change, and “a
silently disarmed destroy guard is worse than a refusal.”
Consequences a rebalancer must be built knowing:
- Every protected deployment is already never-move. The rebalancer gets that for free and must not fight it.
protectedandnever_moveare different questions and must stay two flags.protectedmeans “do not destroy”;never_movemeans “do not relocate automatically.” An operator may well want a deployment that a human can move but a scheduler may not; that is the entire point of the annotation.- The rebalancer’s immobility inventory is
protected ∪ never_move ∪ hard-pinned ∪ (RWO volume whose location the candidate cannot reach), and it should be reportable before anyone builds a mover, because if it covers most of the fleet the mover is not worth building. The brief’s inherited figure (31 of 33 deployments immobile pre---closure) is exactly this inventory, taken once.
3.6 What a node selector is not
Section titled “3.6 What a node selector is not”The scorer must never write a swarm placement constraint for a node it merely
scored. ce ent/schema/deployment.go:104-107 states the rule: “under swarm
ONLY an explicit --node lands here… SHC must never add a placement constraint
the operator did not ask for.” A scored placement is SHC’s opinion, not the
operator’s request. §4.3 is where that bites.
4. (c) Where placement is decided
Section titled “4. (c) Where placement is decided”4.1 Install-time placement should ship before any rebalancer. I agree with the brief, for four reasons, one of which is not the obvious one.
Section titled “4.1 Install-time placement should ship before any rebalancer. I agree with the brief, for four reasons, one of which is not the obvious one.”- It moves no data. The worst outcome of a bad first placement is a deployment on a node the operator would not have picked. The worst outcome of a bad rebalance is a stopped app, or worse. These are not comparable risks.
- It exercises the entire scoring stack under load. Every install runs it. A rebalancer runs it rarely, which is the worst possible way to gain confidence in a decision function.
- The transport already exists and is in production use (§1.3). This is the
reason I did not expect: install-time placement is not a new capability, it is
supplying an argument to a capability that already ships.
shc install --node otherproves the whole path today. - It produces the corpus the rebalancer needs. After a few hundred scored installs there is a record of what the scorer chose and what happened next. Without that, the first rebalance is the first time anyone finds out whether the weights are sane.
The argument against, stated fairly: install-time placement delivers no benefit on a single-node estate, and a scorer that only ever has one candidate is untested by its own production use. That is real. The mitigation is that the scorer is a pure function and its tests are cheap; the production exercise is a bonus, not the proof.
One more argument against, which I take more seriously: install-time
placement changes where things land, and “where things land” is currently
predictable (“wherever I ran the command”). An operator who has built habits
around that predictability will experience the scorer as the machine putting my
app somewhere weird. That is a documentation-and-explain-output problem, and
§4.4 is the answer, but it is a real cost and it is why shc placement explain
is in phase 1 rather than phase 3.
4.2 The exact seam, and the wrong door
Section titled “4.2 The exact seam, and the wrong door”The right door. ce modules/stack/service.go:409-436, Service.Install.
After resolveInstallTarget and after StatedNodeID is captured, before
CheckRuntimeMatchesNode:
if req.StatedNodeID == "" && target.NodeID == "" { target.NodeID = <scorer's choice> // routing only}Everything downstream then behaves as it does for --node today:
makeRoutingStackInstaller dispatches, StampPlacement stamps, the anchor and
the pin are written by the intent functions.
★ The wrong door, and it is one line away. Setting req.NodeID instead of
target.NodeID (or letting the existing back-fill at :425-427 do it for you)
writes the scored node into StackRepoRow.PinnedNode
(modules/stack/service.go:865), which PlacementIntent turns into
deployment.pinned_node_id. On compose that is harmless-looking (it is what
already happens). On swarm it manufactures a hard pin the operator never
asked for, which is the exact violation deployment.go:104-107 forbids and
which StatedNodeID was introduced to make expressible.
The fold at :425-427 currently has two cases (--node given / not given) and a
scorer needs a third (derived). Either add req.ScoredNodeID alongside
StatedNodeID, or make the back-fill three-way. Do not reuse req.NodeID.
That is the highest-value sentence in this section; the same class of fold
destroyed the same distinction once already, and the comments at :417-421 and
:857-864 are the record of it.
4.3 The swarm asymmetry: install-time placement is a compose feature
Section titled “4.3 The swarm asymmetry: install-time placement is a compose feature”On swarm, SHC should not choose a node. Swarm’s scheduler exists, and at ce HEAD
SHC now feeds it the reservations it needs to make an informed choice (§1.2).
Overriding it means SHC and swarm both scheduling, and the doctrine already
says SHC loses that argument (“swarm owns placement; SHC owns intent”,
shc modules/cluster/drain.go:135-141).
So: the scorer runs when SchedulerOwnedRuntime(runtime) is false. On swarm
it does nothing, and the honest framing is that SHC’s contribution to swarm
placement is the reservation figures, not a node choice.
Given both estates are 100% compose (inherited) and no swarm-runtime install has ever been verified end to end (inherited), this is not a limitation. It is a statement that install-time placement lands exactly where the fleet is.
4.4 shc placement explain is phase 1, not a nicety
Section titled “4.4 shc placement explain is phase 1, not a nicety”The owner named “an audit trail answering why did it move this” as a prerequisite. The same requirement applies to why did it put this here, and it is much cheaper to build against the pure function than to reconstruct later.
$ shc placement explain gitlabtarget: node-b (score 612)
candidate mem-free cpu-free deps tenant app score note node-a 2.1 GiB 3200m 7 4 0 388 node-b 9.4 GiB 6400m 3 1 0 612 <- chosen node-c unknown unknown 4 2 0 411 memory axis abstained (node.memory_bytes is NULL) node-d excluded: node.labels.ssd==true (no node in this fleet carries any labels; see `shc node label`)Two requirements that are easy to get wrong:
- The explain output must be a replay of the recorded decision, not a re-derivation. A re-derivation run a day later reads a different tally and disagrees with what happened, which destroys exactly the trust the feature exists to build. Persist the score vector and the candidate set with the decision; §2.6’s determinism is what makes the persisted form small (the inputs, not the outputs).
- Exclusions must name their cause, including derived anti-affinity exclusions (§3.3) and abstained axes (§2.7). A candidate that vanished with no stated reason is how operators conclude the scheduler is haunted.
4.5 The kill switch, designed in
Section titled “4.5 The kill switch, designed in”placement.scoring: off | on resolved through the ordinary cascade, and off
means exactly today’s behaviour: the request runs on the daemon that received
it. Not “score but ignore”: do not score. The distinction matters because a
kill switch that still runs the code is a kill switch that can still be crashed
by the code.
Because the seam is one assignment inside one if, this is genuinely cheap now
and genuinely expensive later, which is the owner’s own instinct applied to the
scorer rather than to the mover.
5. (d) Bin-packing vs spreading
Section titled “5. (d) Bin-packing vs spreading”5.1 SHC’s default is SPREAD
Section titled “5.1 SHC’s default is SPREAD”Four reasons, in ascending order of force:
- SHC has no HA. The owner’s own framing: “HA is desirable but optional”
(inherited,
atomic_move_design). A packed node’s failure takes down everything on it and nothing fails over. Density’s payoff is fewer nodes; its cost is a bigger crater, and SHC has no parachute. - Recovery is expensive here. Moving a deployment off a failed or overloaded node is a backup-and-restore for every deployment on both estates (zero block volumes, zero storage providers, inherited). Spread is cheap insurance against an operation that is measured in minutes per stack.
- Bin-packing needs an admission gate to be safe, and there isn’t one.
Packing deliberately drives nodes toward their limit; the three gates that
would catch overshoot have zero callers and their own decision doc
recommends
warn, notrefuse. Packing without admission is packing without a floor. - The customer is a self-hoster. The population that runs SHC has already bought the hardware. Density optimises a cost they have mostly already paid, while blast radius is a cost they pay every time.
Spread. Default. Not configurable off in phase 1.
5.2 The knob, when it exists
Section titled “5.2 The knob, when it exists”placement.strategy: spread | pack, one key, resolved through the ordinary
5-level cascade: global → tenant → environment → stack → deployment
(shc docs/reference/quota-model-state.md:110, docs/reference/cli/config.md:72).
There is no node level in that cascade and this proposal does not ask for one.
The owner’s ruling is structural and correct: node is an infrastructure axis,
not a config scope, and the cascade is the tenant/environment/stack tree
(inherited, memory reservation_ratio_ruling).
And the ruling composes with this key more neatly than it does with the
reservation ratio, which is worth stating. placement.strategy is resolved at
the scope of the deployment being placed, and it describes how SHC should
treat the fleet on that deployment’s behalf. “This tenant’s workloads should
spread; that tenant’s dev environment can pack” is coherent, resolvable, and
needs no node scope to express. A per-node strategy would be incoherent anyway:
strategy is a property of the decision, and one decision ranges over all
nodes, so a per-node value would have no single reader.
The mechanism is §2.5’s negated weights: one function, one explain output.
5.3 What I am not proposing
Section titled “5.3 What I am not proposing”- No per-node knob, per the ruling.
- No
placement.strategyin phase 1. Ship spread as a constant. A knob shipped alongside its first implementation is a knob nobody has evidence to set, and it doubles the behaviour surface before anyone has seen the single behaviour work. - No third strategy (“balanced”, “cost-aware”). Two opposites and a weight vector already cover the space; a third name would be a preset, and presets are how a policy surface becomes unexplainable.
6. Recommendation
Section titled “6. Recommendation”Phase 1: the scorer, install-time, compose only. Pure function; signals
S1–S5; spread weights as constants; PlacementConstraint reused as phase 1 of
the filter; placement.scoring: off|on kill switch defaulting off for one
release, then on; shc placement explain; drain migrated onto the shared
scorer with count-only weights so its behaviour is provably unchanged. No
data moves. No new schema.
Phase 2: the durable inputs. node.labels (or node-keyed config rows) so
label constraints stop silently matching nothing; a durable free-disk fact
(S6); deployment.never_move in the protected shape; persistence of
placement provenance so pinned_node_id stops being a lie about operator
intent on compose (§1.5).
Phase 3: constraints and anti-affinity. --constraint / --prefer on
install; placement.anti_affinity_group soft by default; derived exclusions in
the explain output.
Phase 4: the knob. placement.strategy: spread|pack, only if an operator
has produced a case that spread gets wrong.
Phase 5 (and only then): the rebalancer consumes the scorer. By this point the scorer has placed hundreds of deployments, its explain output is something an operator reads without surprise, and the immobility inventory (§3.5) is a known number rather than a guess.
What I would not do: ship a rebalancer that carries its own scoring logic “for now.” Two scorers is how the fleet ends up with two answers to where a deployment belongs, and the drain picker is already the first of them.
7. What this will get wrong
Section titled “7. What this will get wrong”Stated plainly, because it will.
- It scores reservations, and reservations are a derivation, not a measurement. The figure is quota × 0.25, where quota is usually the app author’s authored limit. An app that over-declares gets scored as heavy forever; an app that under-declares gets scored as light and then eats the node. The scorer will be confidently wrong about any app whose declaration is wrong, and nothing in the design detects that: the loop from “what it actually used” back to “what we scored it as” does not exist and this proposal does not build it.
- It will pick badly at N=2. With two candidates and one large deployment, any headroom-weighted scorer puts the big thing on the emptier node and the next big thing on the other, which is right; but a third big thing has no good answer and the scorer will pick one and look arbitrary. Small fleets have small numbers of good answers.
- It does not know about disk at all until S6 exists, so it will happily place a 200 GiB app on the node with the most RAM and the least free disk. Given the catalogue’s largest apps are the ones with the largest data directories, this is not a hypothetical.
- It does not know about network topology, location, or egress cost.
node.locationexists on the row and the scorer ignores it. A fleet spanning two Hetzner locations will get placements that are memory-optimal and latency-hostile. - It cannot see co-tenancy through a plug. Two deployments joined by a plug
(an app and its
-dbcompanion) want to be together, and the scorer’s spread terms actively push them apart. The--closurework knows how to compute that relation; the scorer does not consume it. This is the most likely source of a genuinely bad first placement and it should be the first thing measured after phase 1 ships. - The explain output will be read as a promise. An operator who sees “score 612 vs 411” will assume the difference is meaningful. It is a weighted sum of normalised terms with hand-picked constants; the ordering is meaningful and the magnitude is not, and no amount of documentation fully fixes that.
- It changes where installs land, which will be experienced as a regression by anyone whose runbook says “ssh to the node, run the install.” That is a real cost paid by real operators and the kill switch is why it is survivable.
8. Rejected alternatives, with their costs
Section titled “8. Rejected alternatives, with their costs”8.1 Keep pickTarget and just add a memory term. Cheapest possible change,
and it would capture most of the §2.2 benefit. Rejected because it leaves the
scorer inside drain.go, unexported and uncallable from install, which is the
actual defect. The scoring maths is the easy part; the location is the
change. Cost of rejecting: a few hundred more lines than the minimum.
8.2 Wire the three physical capacity gates and call that placement. They
answer “does this fit an empty box,” never “is there room now,” and §2.2
measured that at estate scale they would fire in ~0% of cases. Wiring them is
not placement and would produce a feature that appears to work because it never
refuses. Cost of rejecting: the swarm preflight lane still has nothing to refuse
an unschedulable stack with, unchanged by this document, and still governed by
docs/decisions/2026-08-04-physical-capacity-gates.md.
8.3 Score on live utilization (get_health / StatsCache). Rejected on three
grounds: the cache is per-process and 15 seconds and empty before the first tick
(inherited), so the answer depends on which daemon you asked; live
utilization is not reproducible, which kills the audit trail; and scoring on
current load creates a feedback loop where a node that gets quiet attracts
placements until it is loud. Reservations are stable, replicated, and already
written. Cost of rejecting: the scorer cannot tell a node that is committed
heavy but actually idle from one that is genuinely busy; see §7.1.
8.4 Let swarm do it (adopt swarm as the multi-node path). Swarm’s scheduler
is real, and at ce HEAD SHC finally feeds it reservations. Rejected because
swarm is a second-class runtime, the fabric is the strategic path, the two are
mutually exclusive at shc init with no in-place flip (inherited,
swarm-contention-map.md:467, swarm-quorum-and-health.md), 37 of 73
catalogue apps could not docker stack deploy at all when last measured
(inherited, M27), and both estates are 100% compose. Cost of rejecting: SHC
writes a scheduler it could in principle have borrowed.
8.5 Make the weights config from day one. Rejected: five knobs no operator has evidence to set, each of which must then be honoured forever. Constants first; a knob when someone produces a case. Cost of rejecting: an operator with an unusual fleet has no recourse but the kill switch until phase 4.
8.6 Score at every install including swarm. Rejected per §4.3: it manufactures placement constraints the operator did not ask for and puts SHC in contention with a scheduler that owns the decision. Cost of rejecting: swarm installs keep whatever swarm decides, which is the correct outcome anyway.
9. Rulings owed
Section titled “9. Rulings owed”9.1 Bin-pack or spread as the product default? §5 recommends spread and gives four reasons, but this trades hardware spend against blast radius and only the owner prices both. If the answer is pack, §2.5’s negated weights are the whole implementation, but §5.1’s third reason then becomes load-bearing and the capacity gates need a posture before packing ships.
9.2 May a scored placement be recorded as a pin? Today pinned_node_id is
set for every compose deployment (§1.5), so the column already cannot express
operator intent on the runtime both estates use. Options: (i) leave it, and add
a separate provenance signal a rebalancer reads; (ii) change the compose
semantics so only an explicit --node pins, a behaviour change with a
migration; (iii) accept that a rebalancer must treat compose pins as advisory.
I recommend (i) (it is additive and it does not touch a column three
subsystems already read), but it is a schema decision and belongs to the owner.
9.3 Is anti-affinity soft or hard by default? §3.3 recommends soft because one node is a supported topology and hard anti-affinity makes the second member uninstallable there. An owner who considers co-location of two members a correctness violation rather than a preference would rule the other way.
9.4 Does placement.scoring default on or off on first release? §6 proposes
off for one release, then on. The argument for defaulting on immediately is
that a feature nobody enables gets no production exercise, which is the whole
point of phase 1. The argument for off is §7.7. This is a release-posture call.
10. Prerequisites, in dependency order
Section titled “10. Prerequisites, in dependency order”P1. The shared placement package. Move candidateNode,
candidateTargets, filterCandidatesByConstraints and pickTarget out of
shc modules/cluster/drain.go into a package both drain and install can import.
Blocks everything. First test: drain’s existing behaviour is byte-identical
with count-only weights.
P2. The commitment query. Σ service.cpu_request_millicores × desired_replicas and Σ service.memory_request_bytes × desired_replicas,
grouped by node via deployment_node (anchored rows). No new columns (§1.2).
Blocks S1/S2, i.e. everything that makes the scorer better than pickTarget.
P3. The install seam. A third case in the req.NodeID / target.NodeID
fold at ce modules/stack/service.go:425-427 that can express “derived” without
writing a pin (§4.2). Blocks install-time placement. This is the one place a
mistake is expensive, and the mistake is one character of variable name.
P4. shc placement explain + decision persistence. Blocks operator trust,
which blocks turning the kill switch on.
P5. Node labels with a durable home. Blocks every node.labels.* constraint
outside swarm, which is to say, blocks them on both estates (§3.2).
P6. deployment.never_move in the protected shape. Blocks any rebalancer.
Independent of P1–P5; can be built in parallel by whoever is doing the mover.
P7. A free-disk fact with a durable home. Blocks S6. Lowest priority of the seven and the easiest to defer, but see §7.3 for what deferring costs.
11. Reproducing the measurements
Section titled “11. Reproducing the measurements”All figures in §2.2 come from parsing shc/apps/**/compose*.y*ml with
yaml.safe_load after stripping jinja ({{…}} → 0, {%…%} → “), summing
services.*.deploy.resources.limits.memory per app directory, excluding paths
containing /tests/ and fixtures/. The reservation operand is that figure ×
0.25 (deploy.reservation_ratio, §1.4). The simulation places k apps sampled
without replacement onto 3 nodes, picking min(key, node_index) per app, seed
20260811, 3000 draws. Node size 15.6 GiB is the cpx32 MemTotal from
docs/decisions/2026-08-04-physical-capacity-gates.md §3.3 (inherited; I
did not measure a node).
The validation that the parse reads the right thing: it independently reproduces
that document’s gitlab figure of 16.25 GiB.
What the simulation is not. It assumes apps are placed one at a time in random order onto an empty fleet, which is how a fresh estate fills up and is not how an existing one evolves. It uses authored limits as a proxy for real consumption, which §7.1 already flags as the design’s central weakness. Read the ratios between pickers as the finding; the absolute imbalance numbers are illustrative.
Related
Section titled “Related”shc modules/cluster/drain.go:pickTarget,candidateTargets, the count bumpce modules/cluster/placement.go:PlacementConstraint,NodeSatisfies,HasHardNodePince modules/stack/service.go:409-436: the install seam, and the fold to be careful withce daemon/services_stack_routing.go: the cross-node install dispatch that already existsce modules/app/protected.go: the two-store, fail-closed annotation shape to copyce modules/clone/protection.go: whyprotectalready refusesmovece ent/schema/deployment_node.go: the occupancy set the count should readce ent/schema/deployment.go:96-123:recorded_node_idvspinned_node_idce modules/app/resource_quota.go: where reservations are derived and writtenshc modules/quota/enforcement.go: the unknown-is-not-zero ruleshc docs/decisions/2026-08-04-physical-capacity-gates.md: the three inert gates, still openshc docs/architecture/cluster/swarm-contention-map.md: M27, M33, and the second-class-runtime framingshc docs/reference/quota-model-state.md: the derivation ladder and the five scope tiers