What triggers a move, and how it cannot run away
Status: open design. Automatic rebalance is not built. A move runs only when an operator asks for one.
★ SUPERSEDED IN PART: read
Section titled “★ SUPERSEDED IN PART: read docs/decisions/2026-08-11-rebalancer-v1-foundations.md first”docs/decisions/2026-08-11-rebalancer-v1-foundations.mdfirstThis page is the DESIGN DRAFT. The lane that built from it landed on 2026-08-11 and measured several of its claims differently. Where this page and that decision disagree, the decision is what was measured and this page is kept for its reasoning, not for its facts.
The corrections that matter here:
- NONE of this page is built, deliberately. v1 is placement plus the substrate: no autonomous trigger, no rebalance timer, nothing that moves a live deployment, no failover. The asymmetry is the reason; a bad first placement is undone by the next install, while a bad rebalance is a full stateful cutover whose revert window closes at the commit point.
- What this page’s R1..R6 target contract will be able to READ, once somebody builds it, is now real: the per-node commitment ledger (
shc modules/quota/commitment.go) and the scorer (shc modules/placement). Thecommit_ratioits thresholds are expressed in is computable today.- Its threshold numbers were written against a commitment operand of
quota x 0.25. That operand is nowquota x 1.0, so every ratio in §R1..R6 is 4x what it would have been and the low/high bands need re-deriving before anything gates on them.Everything else on this page stands, including the parts the implementation did not reach; they are named in §8 of the decision.
As written: PROPOSED. Nothing here is built. This is lane 4 of the automatic
deployment-distribution design: the trigger, the safety envelope, the relation
to shc node drain, the per-app standby declaration, and the timer the move
reconciler still lacks. Six rulings are owed (§9). Two claims in the brief this
lane was handed are corrected against HEAD (§1.2), and one measurement puts
a hard ceiling on what the feature can do on the estates that exist today
(§2.6); the ceiling is not a defect and the design does not try to route
around it.
Baselines. shc main c0b009ba5, ce main 6a3dcc5, both read on
2026-08-11. Every figure below is marked MEASURED (I read it in those trees
or computed it over the catalogue in this checkout), RECORDED (a source
comment states a measurement I did not repeat), or PROPOSED (a design
choice with no evidence behind it yet).
The one-line recommendation. Build the evaluator, the decision record and the dry-run planner first and let them run for weeks with no actuator at all; make the trigger a pressure-relief valve, never an evener; make the per-deployment cooldown the outer bound on runaway; and refuse to auto-move any app that has not declared a standby mode.
1. What exists, and what the brief got wrong
Section titled “1. What exists, and what the brief got wrong”1.1 The parts that are already built
Section titled “1.1 The parts that are already built”MEASURED. The move primitive is largely there and it is careful. The
durable node-local journal (ce/modules/clone/move_journal.go), the
daemon-start reconciler (ce/subsystems/impl/move_journal_recovery.go,
priority 73, ModeAny, 15-minute ceiling), the container-observation health
gate (move_health.go) and the byte-shortfall data gate (move_data_gate.go)
all exist, all fail closed, and the destroy guard already refuses a move
against a protected deployment (protection.go).
MEASURED. Nothing schedules. The only node-scoring code in either repo is
pickTarget at shc/modules/cluster/drain.go:512: lowest count of running
deployment rows, ties broken by node id, runtime match preferred. It is
unexported and its sole caller is Drain. No CPU, memory, disk, quota or
affinity term appears in it.
MEASURED. shc node drain is EE-gated. Fleet verbs register through
registerFleetCommands; the CE binary gets refusing stubs
(ce/modules/node/commands/main.go:142-148, code X403028CLIEEO). Whatever
this lane builds inherits that gate, because it is strictly more than a drain.
1.2 Two corrections
Section titled “1.2 Two corrections”CORRECTION 1: commitments are summed, per node, on every metrics tick.
The brief says “totals exist, nothing sums commitments”. MEASURED:
ce/subsystems/impl/scheduling_source.go implements
metrics/collectors.SchedulingSource and is wired at
ce/subsystems/impl/collectors.go:341. Per node it produces CPUCapacity,
MemoryCapacity, DiskCapacity, CPUScheduled, MemoryScheduled and
StackCount. The commitment sum exists and runs today.
It is not usable as-is, for four reasons, each of which is a stage-1 work item:
| defect | evidence |
|---|---|
it sums limits, not reservations (sv.CPULimitMillicores, sv.MemoryLimitBytes in collectScheduled) | docs/decisions/2026-08-04-physical-capacity-gates.md §3 measured the limit operand to be the wrong one, and measured it refusing installs that succeed today |
a stack whose Node is empty is bucketed under the literal key "local", which matches no node row | scheduling_source.go collectScheduled |
| swarm-runtime stacks aggregate cluster-wide rather than per node | same function; correct for swarm, but a mixed fleet gets an un-attributable bucket |
| the output goes only to OTLP gauges | ce/modules/metrics/router.go:13: “there is NO /metrics scrape endpoint; push-only”. No decision path can read the number back |
CORRECTION 2: cross-node live utilization exists on the leader. The brief
says “StatsCache is per-process”. MEASURED:
ce/modules/metrics/collectors/peer_nodes.go is a leader-only fan-in that asks
every peer for its local NodeStats over the cluster mTLS internal API and
merges the answers into the leader’s own cache; it is wired at
collectors.go:368 and fires on the metrics tick, default 2s
(resolveTickInterval, config timing.tick_interval). NodeStats carries
CPUPercent, MemoryPercent, DiskPercent, DiskUsedBytes,
DiskTotalBytes, Load1m/5m/15m and ContainersRunning per node.
What remains true: the cache is in-memory, unpersisted, and empty before the first tick after a restart. An evaluator must therefore refuse to run against a cold cache rather than read zeros as calm.
Related, and softer than the brief states: per-node free disk is not “no
fact anywhere”. ce/modules/clone/preflight_space.go:134 has a real
DiskFreeBytes (Bavail x Bsize) probe used by clone preflight, NodeStats
carries used and total per node cluster-wide on the leader, and a
get_disk_usage internal RPC exists. What is missing is a durable column,
not a measurement.
1.3 Two numbers the metric has to be built around
Section titled “1.3 Two numbers the metric has to be built around”MEASURED, over the 104 apps/**/compose.yaml files in this checkout:
- 94 declare
deploy.resources.limits. Zero declaredeploy.resources.reservations. (A plaingrep reservations:finds one hit, inapps/standalone/ollama; it is inside a comment showing how to request a GPU. Parsing theresources:blocks properly returns nothing.) - 81 carry an
x-shc-scaleblock; of the 74 my parse read cleanly, zero declare abackup:orrestore:window. This agrees with ce’s own comment indaemon/backup_scaler.go(“all are max-only”).
MEASURED, quota derivation at HEAD (ce/modules/app/resource_quota.go:13,
resource_reservations.go:105): limit = quota x deploy.limit_ratio, default
1.0, range [1, inf); reservation = quota x deploy.reservation_ratio,
default 0.25, range (0, 1], with per-axis overrides
deploy.cpu_reservation_ratio / deploy.memory_reservation_ratio. The brief’s
“reservation_multiplier (1.0), limit_multiplier (1.5)” matches neither the key
names nor the defaults in this tree.
The consequence, and it is the honest limit of this whole feature. The
capacity decision says the operand must be the reservation. But no app in the
catalogue declares one. Every reservation in either estate is
quota x 0.25, derived, written into the rendered artifact and then persisted
into service.cpu_request_millicores. So a rebalancer that balances committed
reservations is balancing a policy number the operator invented, scaled by a
constant; arithmetically it is the limits sum times a quarter, carrying no
independent information about what any app needs. That is why the trigger in
§2 requires an observed signal as well, and why §2.2 keeps the two terms
separate instead of blending them into one score.
MEASURED, the persistence that makes the join cheap: ent service already
carries cpu_quota_millicores, cpu_request_millicores, cpu_limit_millicores,
memory_quota_bytes, memory_request_bytes, memory_limit_bytes and
desired_replicas; ent deployment carries recorded_node_id and
pinned_node_id. A per-node committed-reservation sum is one grouped query. No
new column is needed for it.
2. (a) The trigger
Section titled “2. (a) The trigger”2.1 The shape of the decision, before any formula
Section titled “2.1 The shape of the decision, before any formula”A rebalancer can be one of two things, and the choice governs everything else:
- an evener, which drives the fleet toward a flat utilization curve; or
- a pressure-relief valve, which acts only when a node is genuinely in trouble and otherwise leaves an uneven fleet alone.
PROPOSED: SHC builds the valve. An evener moves stateful stacks on a
healthy fleet to improve a number nobody is suffering from, and each of those
moves costs a real outage window (journalPhaseStop opens the downtime window;
MEASURED, move_journal.go). The Kubernetes analogy the owner reached for
breaks here: the descheduler evicts and lets the scheduler re-place, cheaply,
because pods are disposable. SHC has no scheduler to re-place into, and a
relocation is a backup, a ship and a restore. An uneven fleet where every
node is comfortable needs no moves.
The imbalance number the brief asks for therefore still gets defined, but it is a reported figure, not a trigger. See §2.3.
2.2 Inputs, with their honesty flags
Section titled “2.2 Inputs, with their honesty flags”The evaluator reads four things. Each carries a freshness stamp and a
DERIVED | MEASURED flag into the decision record (§3.4), because a threshold
crossed by a derived number and one crossed by a measurement are different
claims.
| input | source | flag |
|---|---|---|
capacity(n): cpu millicores, memory bytes, disk bytes | ent node.cpu_cores x 1000, node.memory_bytes, node.resources.disk_total_bytes; NULL = unknown | MEASURED (probe), may be unknown |
committed(n): sum of reservations of rows recorded on n | grouped query over service joined to deployment on recorded_node_id, Σ cpu_request_millicores x desired_replicas | DERIVED for every app in the catalogue (quota x 0.25) |
observed(n): cpu / memory / disk percent | the leader’s fan-in StatsCache, sampled at timing.tick_interval | MEASURED |
free_disk(n) | DiskFreeBytes per node, durably recorded (stage 1) | MEASURED |
An unknown axis is never a breach. The per-axis unknown guard already
landed in shc/modules/quota/enforcement.go (knowsCPU / knowsMem, which
also read a pointer-to-zero as unknown); the evaluator must use the same
disposition. A node whose memory total is NULL is not a node under memory
pressure; it is a node the fleet cannot answer for, and it is excluded as both
source and target with UNKNOWN_CAPACITY in the record.
2.3 The metric
Section titled “2.3 The metric”capacity(n, axis) per the table abovecommitted(n) Σ over deployments d recorded on n, over services s of d, s.<axis>_request x s.desired_replicascommit_ratio(n, axis) = committed(n, axis) / capacity(n, axis)
sustained(n, axis, w) = the 10th percentile of the observed samples for (n, axis) over window w -- i.e. the level that 90% of the window exceeded
pressure(n) = max over axes of max( commit_ratio, sustained )spread = max pressure(n) - min pressure(n), over active nodesspread is the imbalance metric, and it never triggers anything. It is
what shc node capacity and shc rebalance plan print, what an operator
watches, and what a future ruling could promote into a trigger once it has been
observed on a real fleet for a quarter. Promoting it is a one-line default
change; demoting it after it has caused a move is not.
The trigger is a two-gate test, and both gates must pass.
Gate 1: PRESSURE. A node s is a candidate source only if, for at least
one axis, pressure(s, axis) > high where high defaults to 0.90.
sustained is a windowed 10th percentile, so the “node hit 71% for ten
seconds” case cannot arm the gate at all; 90% of a 30-minute window has to
be above the line. That is the hysteresis, built into the metric rather than
bolted beside it.
Gate 2: BENEFIT. For a candidate deployment d on s and a candidate
target t, all six must hold:
R1 commit_ratio'(t) <= low (low = 0.75)R2 commit_ratio'(s) < high (the move actually relieves)R3 max(p'(s), p'(t)) <= max(p(s), p(t)) - min_improvement (0.10)R4 free_disk(t) >= size(d) x disk_headroom (2.0 -- the heavy strategy stages an archive AND the restored tree)R5 t is active, not draining, not in target cooldown, and t != sR6 every one-hop connected peer of d is already on t, or d has no peerswhere p' is pressure recomputed with d’s reservation subtracted from s
and added to t.
R4 reuses clone/preflight_space.go rather than re-deriving a headroom
rule. Two space calculations that must agree are one calculation.
R6 is the restraint that matters most, and §2.6 explains why it will refuse nearly everything on today’s estates.
2.4 Cadence: three clocks, only one of them new work
Section titled “2.4 Cadence: three clocks, only one of them new work”- Sampler: unchanged. The 2s metrics tick already fills the leader’s cluster-wide cache. Add a bounded ring buffer per (node, axis) so the window quantile is cheap; add no new poll. PROPOSED: a 30-minute window at 2s is 900 samples per series, which at three axes and a ten-node fleet is 27,000 float64, trivial, and deliberately in memory, so a restart clears it and the evaluator refuses until the window refills. A rebalancer that acts on four minutes of history after a daemon restart is a rebalancer acting on the restart.
- Evaluator: a new leader-gated cron,
every 5 minutes(crons.rebalance_evaluate). It writes a plan and a decision record. It never moves anything, in any mode. - Actuator: a second leader-gated cron,
every 15 minutes(crons.rebalance_actuate). It picks at most one approved item and calls the existing move path.
Why 5 and 15, and not tighter. RECORDED in
move_journal_recovery.go: measured crowdsec relocations completed end to end
in 38–51s. A gitlab-class stack is minutes plus a backup. An actuator that
fires faster than the work it starts is a queue wearing a control loop’s
clothes. Splitting evaluate from actuate is also what makes stage 2 shippable
at zero risk; the evaluator is the whole feature minus the danger.
Registration mechanics, MEASURED. Both crons register through
core/registry.CronsAll and are fired by subsystems/impl/cron_runner.go,
which already guards overlapping runs of the same cron (inflight map, skip
warn X503087CRNSKP) and already resolves a per-cron config override under
crons.<name>. Schedules must use the prose form ("every 5 minutes");
schedule.Parse rejects "@every 5m" and the runner then drops the job
silently, RECORDED at ce/modules/backup/crons.go:267, where the same trap
would have deleted a crash-recovery net.
2.5 Hysteresis: six dampers, each against a distinct failure
Section titled “2.5 Hysteresis: six dampers, each against a distinct failure”- The windowed quantile (§2.3). Against transient spikes.
- Dual thresholds, arm at
high0.90, disarm atlow0.75. Against a node oscillating across one line. - Per-deployment cooldown, default 7 days. A deployment that moved is not a candidate again until it expires, whatever the metric says. ★ This is the single most important rule in the document, because it is the only one that bounds the worst case independently of whether the policy is correct: a given stack can move at most once a week even if every other gate is wrong.
- Per-node target cooldown, default 1 hour. A node that just received a stack has metrics that do not yet describe it.
- Settle period, default 30 minutes with no evaluation after any install, uninstall, drain, node join or leave, or daemon restart anywhere in the fleet. A fleet mid-change is a fleet whose numbers lie.
- Plan stability. The actuator executes an item only if the identical
(deployment, target)pair appeared in the last 3 consecutive evaluations, 15 minutes of agreement. A flapping metric produces a flapping plan, and a flapping plan executes nothing. This is the damper that survives a policy bug, because it does not depend on the policy being right, only on it being stable.
2.6 ★ The ceiling: measured movability, and why the honest answer is “almost nothing”
Section titled “2.6 ★ The ceiling: measured movability, and why the honest answer is “almost nothing””RECORDED, in ce/modules/clone/service.go:1408-1418 and the block at
:1200-1225, measured on promanager 2026-08-09:
- keycloak’s one-hop neighbourhood is 12 deployments;
- keycloak’s transitive closure is 31 of 33 deployments;
- before
--closureexisted, 31 of 33 deployments were unmovable by any combination of flags; moveClosureMaxStacks = 8, chosen as a policy statement so--closurestays a relocation flag rather than an estate-migration button.
The automatic rebalancer must never pass --closure, and must never pass
--all-connections either. R6 in §2.3 restricts it to deployments whose
one-hop set is empty or already co-located with the target. On an estate where
one shared provider reaches 31 of 33 rows, that will refuse nearly every
candidate.
State it plainly: on the two estates that exist today, this feature will find almost nothing it is allowed to move. That is the correct behaviour and not a bug to route around. Its value lands in three places instead:
- the decision record, which tells an operator why the fleet is stuck;
“31 rows rejected
CONNECTED, naming keycloak” is a far better artifact than a rebalancer that moved them; - a future fleet of many small independent stacks, where R6 passes;
- the drain path, which shares the machinery and is not restricted the same way because an operator asked for it explicitly.
An honest design says the ceiling out loud rather than widening R6 to look useful. Widening R6 is how a rebalancer relocates thirty deployments nobody asked about.
3. (b) The safety envelope
Section titled “3. (b) The safety envelope”3.1 The kill switch
Section titled “3.1 The kill switch”Three surfaces, because one is not enough, and the split respects the config-versus-state ruling (a setter verb must not store state in config).
rebalance.mode: the policy knob. Scoped config, three values:off(the evaluator does not run),observe(evaluate and record, never act),act. Defaultobservefrom stage 2 onward,actnever by default. Resolved through the ordinary 5-level cascade, so a single tenant can be opted in first.<state>/rebalance.stop: the state-side hard stop. A node-local file. Presence refuses every new move on that node. It is a file for the same reasonmove_journal.gois a file (its own header argues this): it must work when the replicated store has no quorum, and quorum trouble is exactly what a misbehaving rebalancer causes. Checked before each item, not once per pass.shc rebalance stopwrites the file, fans out to every reachable node, and reports which nodes it reached and which it did not. A kill switch that reports partial coverage is the only honest kind. Because the actuator is leader-gated, an unreachable follower cannot start a move while it is unreachable; the residual risk is a follower that later becomes leader, and the message should say that rather than imply totality. Making the stop durable is a separate, explicitshc config set rebalance.mode=off.
★ The switch never interrupts a move mid-phase. It makes the next phase boundary refuse, and the move journal’s existing disposition (revert by default, refuse after teardown) is what runs. A kill switch that aborts a relocation at an arbitrary instant is a data-loss button with a reassuring name.
★ Asymmetry, stated because getting it backwards is fatal: the stop file blocks new moves and never blocks a resume or a revert. A rebalancer halted mid-move still needs its recovery to complete.
3.2 The concurrency cap
Section titled “3.2 The concurrency cap”| knob | default | enforced where |
|---|---|---|
rebalance.max_in_flight | 1 | leader, fleet-wide |
rebalance.max_in_flight_node | 1 as source, 1 as target | the node itself |
rebalance.max_in_flight_tenant | 1 | leader |
A node that would be both source and target of two concurrent moves is refused outright.
The count is taken from the move-journal directory, not from an in-memory counter; the journal is what survives the restart a counter does not.
★ The honest weakness: a fleet-wide count needs a fan-out, and a partitioned node can start a move the leader cannot see. So the per-node cap is authoritative and fail-closed (each node counts its own journal directory and refuses on an unreadable one), and the fleet-wide cap is advisory. With a per-node cap of one and a leader-gated actuator, the true worst case is one concurrent move per node, not per fleet. Design for that number rather than claiming the smaller one.
3.3 The never-move annotation
Section titled “3.3 The never-move annotation”Where it lives: the scoped config cascade, as rebalance.never_move
(boolean), resolvable at deployment, stack, environment, tenant and global.
Not a new column and not a new table; the migration doctrine and the
v0.0.26 brick both argue against a migration for a feature that ships off. The
same cascade already carries deploy.reservation_ratio, and its *Source
reporting is the precedent for naming which scope decided (§3.4).
Setting it true at global scope is a third kill switch, and that is deliberate.
Four conditions imply never-move and cannot be overridden:
pinned_node_idis non-empty; an explicit--nodeis an operator’s placement decision (MEASURED:PlacementIntentwrites it only from an explicit--node);- the deployment is protected;
protection.goalready refuses move outright, fail-closed on an undeterminable answer (X409624CLNPRU); - the runtime is swarm; swarm owns placement;
- the app declares no standby mode (§5).
Fail closed. An unreadable annotation is never-move, matching
protection.go’s disposition exactly.
★ A naming trap worth avoiding: shc protect refuses the operator’s move
as well as an automatic one. rebalance.never_move must refuse only the
automatic mover. They answer different questions and must not collapse into
one flag; the existing overload of protect is visible from here and is
flagged as an adjacent ruling in §9.
3.4 ★ The decision record: the artifact that makes this debuggable instead of spooky
Section titled “3.4 ★ The decision record: the artifact that makes this debuggable instead of spooky”Written by the evaluator, once per pass, including on the pass that decided to do nothing, which is the commonest outcome and the one an operator most wants to interrogate (“why is it not fixing my hot node?”).
decision_id uuidevaluated_at RFC3339evaluator_node the leader that ran the passpolicy every resolved rebalance.* value AND, per key, the scope that set it (precedent: the quota *Source fields)policy_hash so two records can be compared at a glancemetric per node, per axis: commit_ratio, capacity, committed, sustained(window), samples_in_window, samples_above_threshold, threshold, freshness_seconds, DERIVED|MEASUREDspread the reported imbalance figure (§2.3)candidates every deployment on every pressured node, each with: tenant/stack/environment, size_bytes, reservation, observed usage, one_hop_peers[]rejected per rejected candidate, ONE reason code plus its evidence: NEVER_MOVE(scope that set it) PINNED(node) PROTECTED NO_STANDBY(the app declares none) CONNECTED(the peers that would be stranded, by name) COOLDOWN(last_moved_at, eligible_at) UNKNOWN_CAPACITY(axis) NO_TARGET(per target: which of R1..R6 refused, with the two numbers that failed the comparison) NO_BENEFIT(projected p'(s), p'(t), improvement, minimum) TOO_BIG(size_bytes, free_bytes per target)selected (deployment, target) or NONE, with the reason NONEprojection p(s), p(t) before; p'(s), p'(t) after; improvementstability how many consecutive passes have agreed on this itemStorage, split deliberately. One row in the existing hash-chained audit
table per decision, carrying decision_id and the one-line verdict; that
table is append-only with prev_hash / hash (MEASURED,
ce/ent/schema/audit.go), which is the right home for the tamper-evident
claim and the wrong home for a 200-candidate dump. The full record lands as
JSON under <state>/rebalance/decisions/<id>.json with a retention.
Operator surfaces:
shc rebalance plan: run an evaluation now, print the record, change nothing. This is the estate dry-run planner the brief says does not exist, and it is the deliverable of stage 2.shc rebalance why <stack>: the most recent record that mentions this stack, selected or rejected, with the reason code and its numbers. ★ The record must be as complete for a rejection as for a selection. A system that only records what it did cannot answer “why didn’t it”, and “why didn’t it” is the question an operator actually asks.shc rebalance history: decisions and the outcome of each acted item.
4. (c) Drain semantics, and exactly how the rebalancer relates to it
Section titled “4. (c) Drain semantics, and exactly how the rebalancer relates to it”Same machinery below the picker; different machinery above it. The difference is the safety story, so it must be structural rather than a comment.
Shared, unchanged: DeploymentMigrator.Migrate and everything under it:
the move primitive, its journal, the health gate, the data gate, the rollback.
The rebalancer is a caller of the move path and adds no new way to relocate a
deployment. That is not a convenience; a second relocation path is a second set
of gates to keep correct.
Not shared, and the reasons:
shc node drain | the rebalancer | |
|---|---|---|
| trigger | an operator names a node | a metric crosses a sustained threshold |
| scope of one run | every running row on the node | at most one deployment per cycle |
| target contract | any active peer beats leaving the row where it is | a target must be an improvement (R1–R6) or there is no move |
| failure | ends the run, node left draining, per-stack reasons on result.Failed (MEASURED, drain.go:269) | one failed item; nothing is left half-flagged |
| deadline | one budget for the whole run, default 300s via timeouts.drain | per item |
--force | stops rows in place when no target exists | no analogue, ever |
★ --force deserves its own line. Drain’s force flag asks the migrator to
stop a row in place. That is a reasonable answer when an operator is taking a
node away. It is never a reasonable automatic answer to load: “the fleet was
busy so I stopped your app” is an outage nobody requested. The rebalancer must
not gain this flag, and the refusal should be written into the code rather than
left to reviewer memory.
PROPOSED, and small: promote pickTarget and candidateTargets out of
drain.go into a placement package with two named strategies:
StrategyEvacuate (drain’s rule: any active peer, prefer least-loaded, prefer
runtime match) and StrategyImprove (the rebalancer’s: refuse unless the
benefit gate passes). Same candidate construction, two explicit verdicts,
nothing about drain’s behaviour changes. The alternative (one picker with a
mode flag) is how drain’s “any target beats none” rule leaks into the
rebalancer on some future refactor.
Two interlocks between them:
- The rebalancer refuses to evaluate while any node is
draining. Two relocators on one fleet is a fight, and drain is the one that was asked for. Drainflips availability todrainingso that “schedulers stop placing new work on the node mid-flight” (MEASURED,drain.go:170), a comment that today describes a scheduler that does not exist. When this lands the comment becomes true, and honouringavailability != activeas a hard target filter is mandatory. ReusingcandidateTargetsgets it for free.
One efficiency note. candidateTargets calls ListByNode once per
candidate node, per drain (MEASURED). That is fine at drain scale and wrong at
a five-minute cadence over a fleet; the evaluator needs one grouped query, not
N.
5. (d) ★ The standby problem, per app
Section titled “5. (d) ★ The standby problem, per app”5.1 The three-state cutover already exists in this codebase, under another name
Section titled “5.1 The three-state cutover already exists in this codebase, under another name”MEASURED. The backup quiesce bracket is the design lane 4 was asked to invent:
ce/modules/backup/hook_bracket.go:openBackupBracketfires theprehalf and returns a closer the caller defers, so the undo runs on every exit path, and an unarmed bracket closes as a no-op.ce/modules/backup/quiesce_journal.go: journal-then-arm, sha256 filename, an in-processliveQuiesceHolders sync.Mapso a sweep can tell “abandoned” from “in flight”, and a clear only after a verified undo.backup_quiesce_resumecron,every 1 minute, on every node, plus a boot kick fired when the backup service is published.
And the persistent toggles it manages are exactly the standby vocabulary:
postgres and supabase ALTER SYSTEM SET default_transaction_read_only = on
(written to postgresql.auto.conf, surviving container and host restart),
mysql and mariadb SET GLOBAL super_read_only = ON, mongodb db.fsyncLock(),
nextcloud php occ maintenance:mode --on.
★ And its own header states the cost in the right words: “a quiesce is not backpressure; it does not make a client WAIT, it makes it FAIL”. A standby window is an outage that the application’s clients see as 5xx, logged in the app and not in SHC. Any operator-facing text for this feature must say so.
So standby is not a new mechanism. It is the quiesce bracket, generalised, with a journal whose shape is already proven.
5.2 The declaration: two layers, both of which already exist
Section titled “5.2 The declaration: two layers, both of which already exist”Layer 1: app.yaml hooks: a new pre_standby / post_standby pair,
alongside the existing pre_backup / post_backup. Same job-file mechanism,
same bracket object, same journal. For most apps that already quiesce, the
pre_standby job is the existing one; nextcloud’s would be
enable-maintenance-mode, which its app.yaml already names.
Layer 2: x-shc-scale gains a standby: field per service (a fourth
field beside max, backup, restore), meaning “run this many replicas of
this service while the deployment is in standby”. standby: 0 on
mautic-worker is the declaration that stops the worker.
Layer 2 is nearly free, and that is the argument for it. MEASURED:
daemon/backup_scaler.go already scales named services to a window count and
back through app.ScaleService; app.ScaleWindowOps already reads the block
from the rendered compose (post-Jinja, always parseable, RECORDED in
scale_window.go: the source template failed a plain unmarshal for 29 of the
48 apps declaring when that note was written, which is why the render now
preserves the marker); extension_lint.go’s xshcRegistry holds a
closed field set for x-shc-scale with a test that derives the expected
set by walking the Go sources, so a typo’d standbyy: is a refusal rather than
a silent no-op. Adding a field is one registry entry, one checker arm and one
reader.
Why both layers, and this is the honest cost. Layer 2 handles services SHC can see. MEASURED over the catalogue: only 13 of 104 compose files declare a service whose name looks like a background worker (dify, windmill, medusa, monica, pulp, hyperswitch, keygen, mautic, frappe, journiv, plane, and both defectdojo variants). That count undercounts the problem badly: nextcloud’s compose has four services and none is a cron; its cron runs inside the nextcloud container; gitlab-omnibus runs sidekiq inside one container. Name detection cannot find those writers, and neither can any other automatic inspection. That is precisely why the declaration has to be authored, and why layer 1 exists.
5.3 What declaring it means
Section titled “5.3 What declaring it means”An app author who declares standby asserts three things, and the app-development doc must say them in these words:
- after
pre_standbycompletes, no process in this deployment writes to its own data or to any database it consumes; post_standbyreturns it to normal, and runningpost_standbyagainst a deployment that never entered standby is a no-op;- the standby state either survives a container restart, or the app says it
does not, in which case it is not auto-movable. Postgres’s
default_transaction_read_onlysurvives; a stopped worker underrestart: unless-stoppeddoes not.
Current declaration coverage, MEASURED over the 104 app.yaml files in this
checkout: 26 declare pre_backup; 8 declare post_backup; 8 declare
both (nextcloud, uptime-kuma, supabase, postgres, mail, mariadb, mongodb,
mysql). The 18 with a pre and no post are dump-style hooks, not quiesce
brackets. So the paired enter/leave shape exists on 8 of 104 apps today;
that is the adoption baseline, and it is small.
5.4 ★ What happens to an app that does not declare standby
Section titled “5.4 ★ What happens to an app that does not declare standby”Recommendation: it is simply not auto-movable. It appears in the decision
record as NO_STANDBY and never as a selection.
One line of justification, which is the whole argument: a fleet that silently double-runs cron against one live database is far worse than a fleet that moves fewer things.
The pleasant side effect is that the feature advertises its own adoption path.
An operator who runs shc rebalance why gitlab and reads
NO_STANDBY(gitlab declares no pre_standby hook) has been handed the exact
work item, by name, instead of a shrug.
5.5 The -db companion, stated more precisely than the brief does
Section titled “5.5 The -db companion, stated more precisely than the brief does”The brief frames the danger as “moving the app but not the DB double-runs cron against one live database”. Sharpened:
- The move primitive as it exists today does not double-run anything. It
stops the source before recovering the target;
journalPhaseStopis labelled “THE DOWNTIME WINDOW OPENS HERE” (MEASURED). Source and target are never both serving. - Moving an app away from its database is already refused.
X400600CLNCONnames the peers that would be stranded, andpeersOutsideTheMoverefuses a one-hop set that is not closed (MEASURED). R6 in §2.3 keeps the rebalancer on the safe side of that refusal rather than reaching for--closure. - So the double-run risk is specific to a future low-downtime cutover, the finish-forward verb that “is reserved as an operator verb that does not exist yet”, or any design that starts the target before stopping the source.
Therefore: standby is not required by today’s move primitive. It is required the instant anyone implements an overlapping cutover. Declaring it now is cheap and reviewable; retrofitting it after someone has shipped an overlap is an incident. That is the argument for building stage 3 before stage 5, not the other way round.
5.6 The standby leak, and why it needs no new thinking
Section titled “5.6 The standby leak, and why it needs no new thinking”A move killed between pre_standby and post_standby leaves nextcloud serving
a maintenance page forever. This is the same failure the quiesce journal exists
for, and it has the same answer: a standby_journal with the quiesce
journal’s exact shape: journal-then-arm, sha256 filename, a
liveStandbyHolders map, a standby_resume cron every 1 minute on every
node, and a boot kick.
Do not write a third variation. quiesce_journal.go opens with “THIS FILE IS A
DELIBERATE COPY OF stop_journal.go, not a variation on it”; that sentence is
an instruction to the next author, and this is the next author.
6. (e) What the move reconciler still needs
Section titled “6. (e) What the move reconciler still needs”MEASURED, the current state: MoveJournalRecovery runs at daemon start
only. ModeAny, priority 73, deps {container_runtime, ent_client}, a
15-minute pass ceiling, work done in a goroutine so boot is not held open,
returns nil on every path so an unresolvable record cannot stop the daemon.
There is no timer.
MEASURED, its two siblings both have one: backup_quiesce_resume and
backup_stop_resume, both every 1 minute, both on every node, both with a
boot kick and an in-process holder map. The move journal is the odd one out,
and the gap is visible: an interrupted move currently waits for a daemon
restart that may not come for weeks.
PROPOSED: a move_journal_resume cron, built as the sibling rather than as a
new shape.
- Cadence:
every 1 minute, eventually. The quiesce cron’s own argument applies verbatim; every tick a deployment waits is downtime nobody asked for, and the sweep is one readdir of a directory that is empty on almost every node almost always, so the cost of the cadence is a syscall. Ship it atevery 5 minutesfirst (§8, stage 0) and tighten only after a proving run shows zero spurious reverts. Use the prose form;"@every 1m"is silently dropped. - Node-local,
ModeAny, not leader-gated. Same structural reason the boot recovery isModeAny:shc moveforwards to the source’s home node before any phase runs, so the record lives on a node that is very often not the leader. A leader-gated sweep would scan the node least likely to hold anything and report clean.
Five interlocks, and the first two are what make a timer safe where a boot kick needed neither:
- In-process holder map. A
liveMoveHolders sync.Mapkeyed on operation id, so the sweep skips a record a live orchestrator in this process still owns. Without it the timer reverts moves that are running. The boot kick never needed this; a boot proves the old process is gone. - An age floor. Do not act on a record whose
UpdatedAtis newer thanmax(2 x the phase's own timeout, 5 minutes). A young record is not abandoned; it belongs to a process that may simply be slow. Again, only the timer needs this. - Overlap.
CronRunneralready guards overlapping runs of the same cron. Inherit it; do not add a second guard. - The 15-minute pass ceiling stays, and a pass that hits it leaves the record for the next pass. Unchanged disposition: nothing is abandoned by a timeout.
- The kill switch, asymmetrically. The stop file blocks new moves and never blocks this sweep. A halted rebalancer still needs its reverts.
★ The honest cost of adding the timer: it changes the blast radius of a resume bug from “once per daemon start” to “once per minute”. Interlocks 1 and 2 are what buy that back, and they must exist before the cadence is tightened.
7. Where the policy lives
Section titled “7. Where the policy lives”PROPOSED: the existing scoped-config cascade, no new table, no migration.
rebalance.mode string off|observe|act default observerebalance.never_move bool falserebalance.max_in_flight int 1rebalance.max_in_flight_node int 1rebalance.max_in_flight_tenant int 1rebalance.commit_high float 0.90rebalance.commit_low float 0.75rebalance.observe_high float 0.90rebalance.window dur 30mrebalance.min_improvement float 0.10rebalance.stability_evals int 3rebalance.cooldown_deployment dur 168hrebalance.cooldown_node dur 1hrebalance.settle dur 30mrebalance.disk_headroom float 2.0crons.rebalance_evaluate spec "every 5 minutes"crons.rebalance_actuate spec "every 15 minutes"crons.move_journal_resume spec "every 5 minutes" -> "every 1 minute"Every value resolves through the ordinary five-level walk (global, tenant, environment, stack, deployment). There is deliberately no node scope, per the standing ruling that node is an infrastructure axis rather than a config scope, and the metric does not need one, because a node’s thresholds are fleet policy, not per-node policy.
Rejected: a rebalance_policy ent table. Cost: a schema migration for a
feature that ships off by default. The migration doctrine and the v0.0.26 brick
both say no, and the cascade already carries comparable knobs
(deploy.reservation_ratio) including per-key provenance reporting, which the
decision record needs anyway.
8. Recommended rollout: six stages, each independently valuable and abortable
Section titled “8. Recommended rollout: six stages, each independently valuable and abortable”Stage 0: the reconciler timer. §6, alone, with no rebalancer anywhere. Value: closes a real gap today. Abort: delete one registration. Exit test: a proving run in which a SIGKILL mid-move is recovered by the timer rather than by a restart, and a week of passes with zero spurious reverts.
Stage 1: inputs, honestly. Fix the four defects in
SchedulingSourceAdapter (§1.2), give it a read path that is not OTLP, record
per-node free disk durably, and ship shc node capacity showing per node:
capacity, committed reservation, sustained utilization, free disk, and the
freshness plus DERIVED|MEASURED flag on every figure.
Value: an operator can answer “is this fleet balanced” by hand for the first
time. Abort: it is a read-only verb.
Note: this is the natural moment to settle the still-open
docs/decisions/2026-08-04-physical-capacity-gates.md, same numbers, same
seam, and stage 1 needs the answer anyway.
Stage 2: ★ the decision record and the planner, with no actuator. The
evaluator cron, the policy object, the full record (§3.4), shc rebalance plan
/ why / history. Nothing moves, in any mode.
Value: a continuously-running estate dry-run planner at zero risk, which is
where the thresholds get calibrated against a real fleet instead of against the
guesses in §2.3. Abort: turn off one cron.
Exit test: four weeks of records on both estates, and a review that says the
thresholds would have fired when an operator agrees they should have.
★ This is the stage the whole document exists for. A rebalancer whose
policy was never observed before it was armed is the runaway.
Stage 3: standby declarations, with nothing consuming them.
pre_standby / post_standby, x-shc-scale.standby, the standby journal and
its resume cron, and shc standby enter|leave <stack> as an operator verb.
Declare it on a handful of apps.
Value on its own: an operator can safely quiesce a stack for maintenance, and
the backup path may want the same verb. Abort: declarations are inert without
a consumer.
Gate to stage 4: shc rebalance plan on a real estate must report a non-zero
count of candidates passing every gate including standby. If it reports
zero, stage 4 has nothing to do and should not be built; see §2.6.
Stage 4: the actuator, off by default, one move at a time. Mode act,
max_in_flight 1, the stop file, the cooldowns, the stability requirement.
First estate exposure is rebalance.mode=act scoped to one tenant.
Value: the feature. Abort: the kill switch, which by then has been in the
tree since stage 2 and been exercised in anger.
Stage 5: only if measured to be needed: the low-downtime cutover. The overlapping three-state flip, where standby stops being belt-and-braces and becomes load-bearing. Do not begin until stage 4 has run on an estate for a quarter.
9. Rulings owed
Section titled “9. Rulings owed”- Scope: fabric only, or compose-runtime multi-node too? Both estates are
100% compose today; the repo’s own architecture pages call the fabric the
strategic multi-node path while noting it is a data plane, not a
scheduler, and the two are mutually exclusive at
shc init. The rebalancer as designed acts on compose rows and is runtime-agnostic below the picker, but the answer changes which estate can ever exercise it. - The physical capacity gates
(
docs/decisions/2026-08-04-physical-capacity-gates.md) are still open, and stage 1 needs them settled. The reservation-versus-limit operand question is the same question §2.2 is answering. never_movedefault: opt-in or opt-out? Recommendation: opt-in for the first two estates (nothing moves unless annotated movable), opt-out as the eventual default. It is one default value, separable from all the code.- Does a rebalance move count against a tenant’s usage or metering? It consumes a backup, a transfer and an outage window that the tenant did not request.
- Tier. Drain is EE; this is strictly more. Which tier, and is the planner (stage 2, read-only) worth exposing more widely than the actuator?
- Adjacent, and visible from here: should
shc protectkeep refusing the operator’s move, now that a separate automatic-mover guard exists? The overload is defensible today because there was no other guard; afterrebalance.never_moveit is two questions on one flag.
10. What this design cannot do
Section titled “10. What this design cannot do”- It cannot rebalance swarm-runtime stacks. Swarm owns placement, SHC
stamps no constraint unless
--node, and the brief reports that SHC writes nodeploy.resources.reservationsinto swarm services (which I did not verify), so swarm’s scheduler would have nothing to honour anyway. Swarm rebalancing is swarm’s job. - It cannot move a stack whose connected set is not already co-located, and on today’s estates that is nearly everything (§2.6).
- It cannot know an application’s readiness semantics. The health gate observes containers and healthchecks; it is explicitly not a smoke test, and the data gate is a shortfall detector rather than a correctness check. Both files say so about themselves.
- It cannot promise the reservation it balances is real. No catalogue app
declares one; every reservation is
quota x 0.25, derived. The observed term in §2.3 is the only measured half of the metric. - It cannot act during a partition. The leader-gated evaluator stops; the per-node cap keeps a partitioned node honest; nothing converges until quorum returns.
- It does not provide HA. The owner’s framing was “atomic is required, HA is desirable but optional”. This document delivers restraint around an atomic primitive. Nothing here makes anything highly available.
- It cannot be calibrated from this document. Every threshold in §2.3 and §7 is a guess with a plausible story attached. Stage 2 exists because those numbers have to be earned against a real fleet before anything is armed.