Skip to content
SHC Docs

The storage constraint: what a move costs today, and what would make it cheap

Status: open design. Research only. Nothing here is built.

Lane 3 of the placement/rebalancing research. Research and design only. Nothing here is implemented; nothing here touches an estate.

Every claim is marked MEASURED (I ran it or read it off the source at the commit named), DERIVED (arithmetic on a measured number, with the arithmetic shown), INFERRED (a reading I could not close), or RULING-OWED (a decision that is the owner’s, not mine).

Bases: shc @ 80e0c4441, ce @ 3d2a0ee. Worktrees /home/admin/selfhosted-cloud/reb2-storage (branch docs/rebalancer-storage-lane3) and /home/admin/selfhosted-cloud/reb2-storage-ce (detached at ce main).


Placement is downstream of storage, and storage is downstream of a declaration that nobody has written.

SHC has built four separate storage mechanisms. Three of them have an empty declaration corpus, and the fourth is structurally undeclarable:

mechanismstate of the codedeclarations in the 99-entry catalogue
x-shc-volumes, block-volume binding, fenced, 9 cloud providersBUILT, wired end to end0
x-shc-scale.<svc>.backup, per-service replica windowBUILT, wired (daemon/backup_scaler.go)0
shared filesystem (nfs / cephfs providers, link table, node provisioners)BUILTstructurally 0: Volume.access_mode CHECK admits only `rwo
hooks.pre_backup / post_backup, app quiesce bracketBUILT, wired24 pre / 7 with a full bracket

The consequence, and this is the number that matters:

MEASURED: by construction, move today selects backup_restore for every deployment on both estates, on every runtime. The rebind fast path has a real-world corpus of zero. The ephemeral_only fast path has a real-world corpus of five.

The move machinery is excellent and it is running in its slowest gear with the handbrake on, because the catalogue never told it anything about storage.

Recommendation (§6): do not build a new storage layer. Declare the one that exists. x-shc-volumes on a small, argued subset of stateful apps (not all 74) is the cheapest thing that converts any move from a byte copy into an API call. Shared filesystem is rejected on this substrate for a reason that is arithmetic, not taste: on Hetzner, CephFS needs block volumes underneath it, so it costs strictly more than block volumes and adds a cluster to operate.


1. (a) The cost of a move today, per class

Section titled “1. (a) The cost of a move today, per class”

1.1 The three strategies, and which one fires

Section titled “1.1 The three strategies, and which one fires”

ce/modules/clone/move_strategy.go::SelectMoveStrategy reduces a per-mount classification to one of three strategies. MEASURED (source read):

any mount NotRebindable -> BACKUP_RESTORE (MOVE-6, fail-closed)
no persistent mount && runtime==swarm -> EPHEMERAL_ONLY
no persistent mount && runtime!=swarm && move -> BACKUP_RESTORE (MOVE-5)
all mounts Rebindable && runtime==swarm -> REBIND
any CrossProvider -> BACKUP_RESTORE (copy-then-swap)

Two gates collapse this to a single answer on the live estates.

Gate 1: MOVE-5 kills every fast path on the compose runtime. relocateBlocked := nodeChanging && !swarm. A node relocation is only a placement re-render on swarm. MEASURED: on a compose-runtime node, every move to another node is backup_restore, including whoami.

Gate 2: with no volume rows, every persistent mount classifies NotRebindable. Trace through ce/daemon/clone_move_adapters.go::ClassifyMountDecision for a mount with no x-shc-volumes binding (i.e. all of them):

  1. repo.AttachmentNodeFor(...) finds no row → volumeID="", currentNode="", location="", providerProven=false.
  2. provider falls back to the deployment’s --storage stamp, defaulting to "local".
  3. targetStorage is empty (no --storage override) → the cross-provider arm is skipped.
  4. isSharedFSProvider("local") is false → the shared-FS arm is skipped. (And even if the operator stamped --storage nfs, !in.ProviderProven returns NotRebindable; the stamp is a claim, not evidence.)
  5. in.Location == "" → the location-scoped block arm is skipped.
  6. Final arm: TargetNode == CurrentNode && CurrentNode != ""; CurrentNode is "", so this is false → MountClassNotRebindable.

MEASURED, deterministic, no live probe needed. With zero x-shc-volumes in the catalogue there are zero volume rows, and with zero volume rows every persistent mount on every app returns NotRebindable. The whole runRebindMove orchestration, the explicit Rebind primitive, the lease, the nine cloud volume providers, all correct, all unreachable.

Method (reproducible; see §10): parsed all 99 apps/*/app.yaml + compose.yaml (jinja stripped), classified every compose volume entry with a Python port of ce/core/volume/resolve.go::DetectMountType, then applied ce/daemon/clone_move_adapters.go::PersistentMountsFromCompose’s persistence rule.

Cross-check: my classifier independently returns 74 of 99 stateful, matching the number recorded from the earlier session and the docs/rebalancer-fork count of apps declaring a volumes: block. Two independent derivations agreeing is the reason I trust the rest of the table.

MEASURED: mount-type census over all 99 catalogue entries:

mount class (SHC’s own MountType)count
DATA (persistent; becomes <deployDir>/mounts/<n> or a swarm named volume)362 references / 211 distinct
FILE (regenerated at render)15
RUNTIME (scratch)7
SYSTEM (absolute host bind)7
tmpfs / anonymous1

MEASURED: distinct DATA mounts per app:

distinct DATA mounts01234569131518
apps252522126321111

Heaviest: greenbone 18, wazuh 15, mailcow 13, matrix 9, pulp 6, plane 6, gitlab 5, highlight 5, librechat 5.

MEASURED: the “cheap” 25 is not 25. Of the 25 entries with no DATA mount:

  • 10 declare zero services: the nine external-* adapters plus standalone/s3. Nothing runs; they are plug-only stacks. They are not cheap to place, they are not placeable.
  • 8 are test fixtures (fixtures/*, */tests/apps/*-consumer).
  • 5 are real, placeable, stateless apps: huly-nextcloud, nextcloud-picker, affine-rs, gitlab-huly, identity-broker, every one a single-service bridge or sidecar.
  • 2 remainder are fixture/consumer overlaps already counted.

MEASURED: the ephemeral_only fast path, on a swarm estate, applies to 5 non-fixture apps. On a compose estate it applies to none.

Phase order for Move, read off ce/modules/clone/service.go:537-686 (MEASURED):

preflightStagingSpace → backupPhase → stopPhase → measureSourceTrees
→ recoverPhase (ship + restore + bring-up) → verifyPhase → dataPhase → uninstallPhase

The capture runs while the source is live (each app’s own quiesce bracket runs inside it). The outage window is stop → recover completes.

Cost 1: the archive is written to the same disk as the data

Section titled “Cost 1: the archive is written to the same disk as the data”

preflight_space.go: the operation stages a full archive under FilesystemProvider.StagingDir() (= <StateHome>/staging) before anything moves. MEASURED from the earlier binding-model analysis and confirmed in this file: source and target must each hold dataset + archive simultaneously. Combined with the MEASURED estate fact that no node on either estate has a second disk (one 610.4 GB sda, sda1 ext4 at /, six nodes), this is a hard ceiling: a deployment larger than roughly 45% of the node’s free disk cannot be relocated at all. The gate refuses with X507632CLNSPC; margins are estimate + estimate/10 + 64 MiB, doubled for a multi-member (--closure) move because the per-member archives and the group .bkg coexist.

Cost 2: the wire format is base64 inside JSON

Section titled “Cost 2: the wire format is base64 inside JSON”

ce/daemon/services_clone_remote.go::shipBakToPeer streams the archive to the target as a sequence of clone.recover_chunk RPCs. Each chunk is cloneRecoverChunkBytes = 2 MiB raw, base64-encoded into a JSON envelope, POSTed synchronously, one round trip per chunk, capped by internal_rpc’s 4 MiB body limit.

I reproduced exactly this shape (2 MiB read → base64 → json.Marshal → HTTP POST → json.Unmarshal → base64 decode → WriteAt) against a loopback server on this 48-core box with tmpfs on both ends (scratchpad/shipbench). No network, no TLS, no disk seek, no tar, no restic. That makes the result a ceiling, not an estimate.

MEASURED (5 runs, 512 MiB each):

runwire inflationthroughput
11.33×69.8 MiB/s
21.33×65.2 MiB/s
31.33×68.8 MiB/s
41.33×62.3 MiB/s
5 (GOMAXPROCS=2)1.33×61.6 MiB/s

Inflation is exactly 4/3 and deterministic. Throughput barely moves when cores are cut from 48 to 2; it is bound by the synchronous request-per-2-MiB round trip, not by CPU. That matters: it will not improve on bigger nodes, and it will degrade over a real mTLS link.

DERIVED: ceiling wall-time for the ship leg alone, at 65 MiB/s:

datasetship leg (ceiling)
1 GiB16 s
8.44 GiB2 min 13 s
100 GiB26 min
250 GiB66 min

The ship leg is one of four legs. Total bytes touched per move ≈ 3× the dataset (read tree → write archive; read archive → ship; receive → write → untar), before the two MeasureDeploymentTree walks the gates run.

Cost 3: the one real dataset size I could find

Section titled “Cost 3: the one real dataset size I could find”

MEASURED (recorded in ce/modules/backup/phases.go, from a live capture): nc on 2026-08-10 held maintenance:mode --on from 04:04:38 to 04:20:01 (15 min 23 s) around an 8.44 GB archive. That fix moved the un-quiesce earlier so the backup window shrank; it does not shrink a move window, because a move keeps the deployment down until the target is serving.

MEASURED (docs/proposals/2026-08-07-backup-schedules.md): openobserve ≈ 5 GB; timeouts.quiesce_hold defaults to 300 s, and a bracket that overruns it escalates to a full stop-and-retake.

Cost 4: nine tenths of the catalogue’s byte cost is unmeasured, and I cannot measure it

Section titled “Cost 4: nine tenths of the catalogue’s byte cost is unmeasured, and I cannot measure it”

I have two dataset sizes for 74 stateful apps. NAMED LIMIT: no estate contact, so no du and no shc backup ls histogram. The honest ordinal classification is below, and it is INFERRED from what each app stores, not observed:

classapps (examples)expected order
config-onlyuptime-kuma, shlink, vaultwarden, kuttMB
DB-and-indexmost of the 74100 MB – 10 GB
repo / registry / artifactgitlab, gitea, pulp, minio, dependency-track10 – 500 GB
mediaimmich, jellyfin, peertube, nextcloud100 GB – TB

What would close this, and it is one command per node: a du -sb over <StateHome>/deployments/*/*/* on each node of both estates, which is exactly what MeasureDeploymentTree already does. That is an estate action and is therefore outside my mandate, flagged as owed work, not done.

★★ D-1 (P1, latent): the data gate is blind on the swarm runtime.

dataPhase refuses to destroy a source unless the target’s deployment tree holds ≥25% of the source’s bytes. MeasureDeploymentTree walks app.DeploymentDir(...) (<StateHome>/deployments/<tenant>/<stack>/<env>) and nothing else (MEASURED, ce/modules/clone/filesystem_local.go:443).

On the swarm runtime an unbound DATA mount is deliberately not rewritten to a host bind; it stays a docker named volume (ce/modules/app/install_runner.go:4088-4100), so its bytes live under /var/lib/docker/volumes/, outside the tree the gate measures. Both sides therefore measure small and similar, the ratio lands near 1.0, and the gate passes with full marks, for a deployment whose real data it never looked at. The gate’s own header names this class (“52 KB / 91 KB / 121 KB nightlies for multi-gigabyte swarm stacks”) and it catches it only when the source tree is large, i.e. on compose.

Mitigating (and a CORRECTION to the recorded ground truth): the named-volume capture gap is CLOSED at ce HEAD. ce/modules/backup/namedvolume.go + namedvolume_observed.go discover unbound named volumes, capture each under its own post-resume pause bracket, refuse the whole backup on an unexplained empty read (X500451SWMEMP), and record per-volume reads in the manifest. The memory note “swarm backups are metadata only” is stale. So is the warning string in SwarmNamedVolumeBackupWarnings (“its data will not travel with this operation”), which still fires on every swarm move.

What is not closed is the gate. The capture may now be correct while the gate that licenses destroying the source still cannot tell.

PROPOSED fix (not implemented): give FilesystemProvider a second measurement (the deployment’s named volumes, via the same SwarmVolumeCapturer seam the capture already uses) and have measureSourceTrees / dataPhase sum tree + named volumes. Uncertainty must stay a refusal, per the file’s own disposition.

D-2 (P3, cosmetic-but-misleading): SwarmNamedVolumeBackupWarnings contradicts namedvolume.go. One of them is wrong on every swarm move today.


2. (b) What would make the bytes cheap to move

Section titled “2. (b) What would make the bytes cheap to move”

2.1 Candidate A: block volumes (x-shc-volumes)

Section titled “2.1 Candidate A: block volumes (x-shc-volumes)”

What exists. Everything except the declaration. DaemonVolumeBinder.Bind (MEASURED, ce/daemon/volume_binder.go:94-280) runs EnsureVolume → Provision → relocate-if-attached-elsewhere → BumpLeaseAndAttach (single monotonic lease writer) → provider.Attach → node-side EnsureMounted (mkfs-if-empty, mount, fstab) → mark attached, and returns the host mountpoint <deployDir>/mounts/<name>, byte-identical to the host-bind path, so ApplyPrep, backup, restore and the compose render need no change. RebindVolume is the explicit detach→lease-bump→attach→lease-gated-mount primitive the move fast path already calls. Nine providers ship (hcloud, aws-ebs, gcepd, azure-disk, cinder, dovolume, linode-volume, vultr-volume, cloudstack-volume).

What a move becomes. stop → cloud detach → cloud attach → mount → re-render → start. Zero bytes on the network. The staging-disk ceiling of §1.3 disappears for the bound mounts. runRebindMove already rolls back volume-by-volume on a failed target start.

What else it unlocks, already built:

  • ce/subsystems/impl/volume_failover.go: leader-only automatic failover-rebind. Off by default (storage.volumes.auto_failover), gated on a proof of old-node fencing (attachment lease expiry + ≥45 s dwell, or a STONITH confirm; the default NoopStonithAgent never confirms), never crosses node.labels["az"]. MEASURED: it fires for nothing today because zero apps declare a volume.
  • ce/modules/app/swarm_placement.go::ApplyVolumePlacement: pins an RWO volume-backed service to its attachment node and clamps replicas: 1. This is a placement input the scheduler would have to respect, not fight.

★★ The hard constraint nobody has costed. MEASURED (Hetzner docs, this session): a volume is 10 GB – 10 TB, single-attach, and a server takes at most 16 volumes.

That number is load-bearing and it kills the naive design:

  • The catalogue declares 211 distinct DATA mounts. One block volume per mount caps a node at 16 mounts. greenbone alone declares 18 and would not fit on a node by itself.
  • DERIVED: bitspur compose01 runs 128 containers across dozens of deployments. Even at one volume per deployment (mount the volume once and put that deployment’s whole mounts/ tree inside it) a node’s budget is 16 stateful deployments. The recorded promanager shape (31–33 deployments on one node) already exceeds it.

RULING-OWED R-3.1: block volumes are a per-deployment escape hatch for the few big stateful deployments, not a catalogue-wide storage layer. The design must pick the deployments, and the budget is 16 per node.

Money. MEASURED, with a caveat I will not paper over: third-party sources disagree on the current Hetzner volume price: €0.044/GB/month net (quoted as checked 2026-08-03) versus €0.057/GB/month (quoted as an April-2026 increase). RULING-OWED R-3.2: confirm against an actual invoice before any number is put in a plan. The shape of the bill at both ends:

scenarioGB€0.044€0.057
6 nodes × 1 × 100 GB deployment volume600€26/mo€34/mo
12 volumes × 100 GB (2 nodes at budget)1 200€53/mo€68/mo
96 volumes × 100 GB (6 nodes at the 16 cap)9 600€422/mo€547/mo

DERIVED: at €0.044, one 100 GB volume ≈ €4.40/month ≈ 10% of a cx53’s monthly cost. For the handful of deployments where a move currently means a 25-minute outage, that is cheap. At 96 volumes it is a second infrastructure bill, which is the argument for the budget in R-3.1.

Moving parts added: one x-shc-volumes block per declared app; a storage.volumes.provider estate config; a hcloud connection (MEASURED from the binding-model analysis: both estates already carry exactly one volume-capable hetzner connection, so the substrate is present and configuration-free); nothing new to operate. Backups, restore, render and the sandbox are unchanged.

New failure mode, stated plainly: a bound RWO mount becomes a hard placement constraint. ApplyVolumePlacement pins the service and clamps it to one replica; the volume attaches only in its own Hetzner location. A scheduler that ignores this produces Pending tasks, not errors.

2.2 Candidate B: a shared filesystem (NFS / CephFS)

Section titled “2.2 Candidate B: a shared filesystem (NFS / CephFS)”

What exists. More than expected, and none of it usable.

  • Providers: ce/modules/storage/providers/{nfs,cephfs,local}; managed bring-up via shc storage nfs enable (real: leader-gated provider CRUD + /internal/storage/nfs/enable-server and enable-client dispatched over the cluster mTLS peer transport) and shc storage cephfs enable (records a provider row; the ceph cluster itself is operator-run, with a full modules/storage/ceph cephadm/mon/mgr/mds/osd driver present).
  • storageapi.ProviderType.IsShared() is a single total authority guarded by a source-parsing exhaustiveness test. Both move classifiers, the node-support probe and the restore-time --storage resolver all go through it.

Why it is dead code. Three independent blocks, each sufficient on its own:

  1. MEASURED: the volume table forbids the row. ent/schema/volume.go:40: Check: "access_mode IN ('rwo', 'rox')". ValidateBindings explicitly refuses access: rwx with X400328VOLACC, and says why: absorbing rwx into rwo would make swarm stamp a node pin and a replica clamp the author asked the opposite of. So no shared-FS mount can ever be providerProven.
  2. MEASURED: ClassifyMountDecision therefore always answers NotRebindable on the shared-FS arm: if !in.ProviderProven { return MountClassNotRebindable }. Both shared-FS arms of both classifiers are unreachable.
  3. MEASURED: nothing routes deployment bytes onto a provider anyway. app.DeploymentDir is unconditionally filepath.Join(stateHome, "deployments", tenant, stack, environment) and panics on an empty stateHome. paths.MountsUnder(deployDir) is unconditional. storageapi.Provider.BasePath() has zero non-storageapi callers (verified by grep across modules/, daemon/, cli/). The --storage stamp is, in the classifier’s own words, “a label over bytes that are on local disk”.

The --driver local --opt type=nfs observation is correct and does not help. It is stock docker with no plugin. But SHC does not create docker volumes for DATA mounts on compose at all; it rewrites them to host binds under <deployDir>/mounts/. MEASURED: the only place ce reads driver_opts.device is modules/backup/namedvolume.go:190, on the swarm named-volume capture path. Adopting the NFS local-driver shape would mean changing the compose renderer’s DATA-mount strategy, i.e. it is not a one-line adoption of a stock feature.

Cost on this substrate: this is the decisive part.

  • NFS: needs a server. On the current estates that is one of the six nodes, on its own 610.4 GB sda. DERIVED: this converts every stateful deployment’s storage into a dependency on one node, strictly worse availability than today, where a node failure loses only its own deployments. Hetzner offers no managed low-latency NFS in the cloud product; Storage Boxes are CIFS/SMB/SSHFS and are not a POSIX substrate a database will run on.
  • CephFS: needs OSD block devices. MEASURED: no node on either estate has a second disk. ce/modules/storage/ceph/osd.go::Add takes a device argument and shells ceph orch daemon add osd <host>:<device>. To run Ceph here you would first buy Hetzner block volumes to feed the OSDs, then run 3× replication on top.

DERIVED, and it settles the fork: on Hetzner, a shared filesystem costs ≥3× the block-volume bill (replication factor) plus the block volumes themselves plus a Ceph cluster to operate. Shared FS is a strict superset of the cost of Candidate A and delivers a property (rwx) that the catalogue does not ask for.

Which apps would a shared FS actually suit? Honestly: very few. The catalogue is single-writer by construction. MEASURED: 65 of 99 apps declare x-shc-scale, and every one of the 65 has a max: 1 somewhere. rwx buys nothing for a service capped at one replica. The genuine rwx candidates are blob trees behind a scale-out front (nextcloud files, immich uploads, peertube media), and for all three, object storage (the already-shipped s3 plug) is the upstream-supported answer, not a POSIX share. nextcloud’s compose already carries a complete OBJECTSTORE_S3_* block behind {% if integrations.s3 is defined %}.

2.3 Candidate C: put StateHome itself on a share (considered, rejected)

Section titled “2.3 Candidate C: put StateHome itself on a share (considered, rejected)”

MEASURED: paths.resolveStateHome honours SHC_STATE_HOME, so an operator could point the whole state home at an NFS mount and make every deployment directory shared with no code change at all. It is the cheapest possible implementation of “shared storage” and it is the wrong one:

  • StateHome also holds the rqlite data, the vault, the move journal (ce/modules/clone/move_journal.go, deliberately a node-local file so it needs no migration), the staging dir and per-node fstab-managed mounts. Sharing it makes every node-local durability assumption false at once.
  • The move journal’s whole design premise (a durable node-local record written ahead of the act) is violated.
  • Cost: near zero to try, catastrophic and hard to reverse when it fails.

REJECTED. Recorded because it is the first thing anyone will suggest and it needs a written reason, not a shrug.

2.4 Candidate D: app-native move (already in use, under-recognised)

Section titled “2.4 Candidate D: app-native move (already in use, under-recognised)”

MEASURED: gitlab excludes gitlab-data, gitlab-config, gitlab-logs from the mount snapshot and instead runs hooks.pre_backup: gitlab-backup-create (a native gitlab-backup dump into the gitlab-backups mount) plus hooks.post_restore: gitlab-backup-restore which does gitlab-ctl stop puma; gitlab-ctl stop sidekiq, replays, and waits for puma.

That is a per-app move path that is already declared, already exercised, and sidesteps both the byte cost (a dump is far smaller than a gitaly tree) and the consistency problem (the app’s own tool guarantees it). MEASURED: 28 apps already declare backup.yaml exclude.mounts covering 60 of the 211 DATA mounts (28%): caches, brokers, logs and, in gitlab’s case, the entire live tree.

This is the cheapest lever in the whole lane and it needs no new mechanism. For every app whose upstream ships a dump/restore tool, the bytes that move are the dump, not the tree.


MEASURED. The integration table (ce/ent/schema/integration.go) records (tenant, provider_deployment_id, provider_socket_name, consumer_deployment_id, consumer_plug_name, connection_values, status) with a UNIQUE index on (tenant, consumer_deployment_id, consumer_plug_name). app.Service. GetConnectedStacks walks it in both directions and resolves each peer in its own scope (a cross-tenant keycloak@master provider comes back under master, not the consumer’s tenant).

The clone module already consumes it as a transitive closure: --closure runs BFS to a cap of 8 stacks, and its own test fixture is exactly the gravity shape: license-broker → keygen → keygen-db.

The data-gravity graph is not missing. It is built, persisted, directed, transitively walkable, scope-correct, and already spent by move.

3.2 The gravity that matters is not app→app, it is app→db

Section titled “3.2 The gravity that matters is not app→app, it is app→db”

MEASURED: SHC has no -db companion convention in code; grep for "-db" across both repos returns one unrelated hit in the caprover adapter. A -db companion is an operator-created deployment joined by an integration edge. So the relationship is exactly and only the integration edge, which is the thing that already exists.

3.3 Whether co-location is a soft score or a hard constraint depends on the fabric posture

Section titled “3.3 Whether co-location is a soft score or a hard constraint depends on the fabric posture”

This is the part I did not expect and it changes the design.

MEASURED (ce/modules/app/fabric_render.go, ce/modules/fabric/{posture,network}.go): cross-node integration on the compose runtime is carried by the SHC fabric: a per-node local bridge plus a WireGuard shc0 overlay, with the renderer writing pinned ipv4_address per service and extra_hosts on the consumer side of every cross-node integration, preserving the same alias strings swarm would have resolved via DNS. And: ApplyFabricTarget returns its input unchanged on every posture that is not shc, “dormancy is structural”. A cluster on posture none, on swarm, or with no recorded fabric row renders byte-for-byte as if the file did not exist.

DERIVED, three regimes:

posturecross-node app→dbgravity is
shcrouted over shc0 + extra_hostsa soft score (latency, failure domain)
swarmswarm overlay networka soft score
unset / none, compose runtimenot routeda hard co-location constraint

NAMED LIMIT: I cannot read either estate’s fabric posture row without estate contact. If posture is unset on a compose estate (which is the default and the zero value) then a scheduler that splits a consumer from its provider produces a deployment that comes up and cannot reach its database. That is not a latency regression, it is an outage.

RULING-OWED R-3.3: before any scheduler places a deployment on a node other than its provider’s, the fabric posture must be read and a non-shc, non-swarm posture must make co-location a hard admission constraint, not a score.

3.4 What a gravity-aware scorer should read, and what it must not

Section titled “3.4 What a gravity-aware scorer should read, and what it must not”

PROPOSED, in the order I would implement it:

  1. Hard constraint: bound RWO volume. If any of the deployment’s mounts is a bound RWO block volume, the node is decided: ApplyVolumePlacement will pin it there anyway. The scorer must not pretend to choose.
  2. Hard constraint: posture (R-3.3). On a non-routed posture, restrict the candidate set to the node holding every integration peer.
  3. Soft score: co-location with providers. Reward placing a consumer on the node already running its transitive closure (GetConnectedStacks, cap 8). Failure-domain and latency both point the same way.
  4. Soft score: anti-gravity for the closure’s own replicas. Not a spread rule: two members of the same consistency group on one node share a failure domain, and the group backup/restore path treats them as one unit anyway. RULING-OWED R-3.4 whether availability or latency wins here; I do not think it can be decided without the owner’s HA posture.

What it must not read: the deployment’s --storage stamp. The classifier’s own comment is the argument: nothing validates it at install, nothing routes mounts by it, “it names where the operator ASKED the bytes to live, never where they ARE”.

Dependency I could not verify: an earlier session recorded the integration table as having 0 rows on both estates. If that is still true, every soft gravity score is computed over an empty graph and silently degrades to “no opinion”. A scorer reading an empty graph must say so, not score 0.


4. (d) The background-worker trap and the standby declaration

Section titled “4. (d) The background-worker trap and the standby declaration”

4.1 First, the good news: today’s move has no double-run risk

Section titled “4.1 First, the good news: today’s move has no double-run risk”

MEASURED: the phase order is backup → stop → measure → recover. The source is stopped before the target is recovered. runRebindMove calls StopDeployment first (LEASE-1) for the same reason.

And on swarm, runEphemeralMove’s re-render inherits Docker’s default update order, stop-first. MEASURED: no app in the catalogue declares update_config, and grep finds no order: start-first anywhere in either repo.

Standby is not required by today’s move primitive. It is required by the zero-downtime move that does not exist yet. This corroborates docs/rebalancer-safety §5.5 from an independent direction.

4.2 How many of the 99 would need a standby declaration?

Section titled “4.2 How many of the 99 would need a standby declaration?”

The honest answer is every stateful app that runs any background writer, and the compose graph cannot tell you which those are.

I tried to derive it. Method: for each app, classify every compose service as routed (has ports/expose), one-shot (restart: no|on-failure), datastore (name/image matches a datastore vocabulary), or background writer.

MEASURED: the derivation fails in both directions:

It undercounts, badly. The worst offenders show zero background services:

appservices in composebackground writers actually present
gitlabgitlab, valkeypuma, sidekiq, gitaly, gitlab-ctl cron, all inside one omnibus container
wordpresswordpresswp-cron, fired by HTTP requests
nextcloudnextcloud, valkey, nextcloud-pgclientcron.php background jobs
matrixsynapse, element, …synapse background tasks, media retention
immichimmich-server, …microservices, merged into the server image upstream

The proof that these are real writers is in SHC’s own files: gitlab’s gitlab-backup-restore.job.yaml runs gitlab-ctl stop puma; gitlab-ctl stop sidekiq; SHC already knows sidekiq is there; the compose graph does not.

It overcounts too. My heuristic flagged elastic, kvs, redpanda, *-pgclient sidecars and jitsi-jicofo as workers. They are not; they are datastores, backup sidecars and signalling services.

MEASURED CONCLUSION: standby cannot be derived. Any attempt produces a set that is wrong on the highest-value apps in the catalogue. The 25 apps my scan flagged with a named worker idiom (dify, windmill, medusa, monica, pulp, mailcow, hyperswitch, keygen, mautic, frappe, journiv, plane, defectdojo ×2, gitlab-runner, openreplay, huly, …) are a lower bound with false positives, not an answer.

The upper bound is the useful number: 74 stateful apps, of which 73 are non-fixture. Assume every one needs a declaration until its author says otherwise.

4.3 The declaration vocabulary already ships (twice) and is unused

Section titled “4.3 The declaration vocabulary already ships (twice) and is unused”

MEASURED:

  • hooks.pre_backup / hooks.post_backup in app.yaml. Census over all 99: post_install 30, post_update 30, pre_backup 24, post_restore 19, post_backup 7, pre_update 2.

    Only 7 apps have a full enter/exit bracket: nextcloud, uptime-kuma, supabase, mail, mariadb, mongodb, mysql. The other 17 pre_backup hooks are dumps, not quiesces: gitlab’s is gitlab-backup-create, and it has no post_backup because there is nothing to undo.

    The 7 brackets are exactly the right shape: nextcloud: enable-maintenance-mode / disable-maintenance-mode; mariadb / mysql / mongodb / supabase: quiesce / resume.

  • x-shc-scale.<svc>.backup: a per-service replica clamp for the archive window, implemented in ce/daemon/backup_scaler.go via docker compose up -d --no-recreate --scale svc=N, with partial-window rollback. MEASURED: 0 of 99 apps declare it. The file’s own header says so: “No app declares a backup/restore window today (all are max-only), so this is a strict no-op in production.”

    MEASURED: 65 apps declare x-shc-scale; all 65 use max: only.

The standby mechanism is not missing. Two of them ship. The declaration corpus is 7 and 0. This is the same pattern as x-shc-volumes, third instance in one lane.

4.4 ★★ The cautionary tale, and why “declared” is not “true”

Section titled “4.4 ★★ The cautionary tale, and why “declared” is not “true””

apps/standalone/mariadb/quiesce.job.yaml is the single most instructive file I read this session. MEASURED, from its own header:

The job declared a quiesce and never performed one, on any estate, ever. It issued SET GLOBAL super_read_only = ON, a MySQL variable MariaDB has never implemented (MDEV-9458, WON’T FIX), copy-pasted from apps/standalone/mysql across an engine boundary nothing in the catalogue checked. The client stopped at the first error, the preceding FLUSH TABLES WITH READ LOCK died with the connection milliseconds later, and every mariadb archive ever taken was read off a live, unfrozen server, stamped as quiesced.

It has since been fixed to FLUSH TABLES WITH READ LOCK; SET GLOBAL read_only = ON; UNLOCK TABLES, and it now proves itself on a second connection, matching the gate value as a string (ON) or a number (1) because MariaDB 12.3.2 and 11.8.8 spell the same boolean differently and @@read_only+0 silently reads a gated server as open.

RULING-OWED R-3.5: a standby declaration MUST carry its own verification, and the verification must run on a fresh connection / process, or the declaration is worse than its absence: it converts “we know we cannot move this safely” into “we believe we can”. A move that spends an unverified standby as permission to start a second writer is a data-loss bug with a green log line.

4.5 What happens to an app that declares none

Section titled “4.5 What happens to an app that declares none”

PROPOSED, and it agrees with docs/rebalancer-safety §5.4:

An app with no verified standby is never selected for a zero-downtime cutover. It is not refused a move; it takes today’s stop-first path, which is correct and always has been. The selection record must name the reason (NO_STANDBY(<app> declares no verified pre_standby)), because a scheduler that silently declines to move the one deployment causing the imbalance is a scheduler nobody trusts.

The corollary is the design’s actual shape: the standby declaration buys downtime, not correctness. Correctness is already bought by stop-first. That reframes the whole item from “blocking prerequisite” to “optimisation with a per-app opt-in”, which is a much better place for it to sit.

4.6 One more constraint on any standby design

Section titled “4.6 One more constraint on any standby design”

MEASURED: timeouts.quiesce_hold defaults to 300 s, and a bracket that overruns it stops the stack, retakes, and restarts it, unattended. Any standby window that could exceed 5 minutes (i.e. any of the media/registry class in §1.3) must either raise that timeout deliberately or accept that its “graceful” standby degrades into a hard stop.


Each of these is a hard number a placement design must satisfy.

  1. MEASURED: Zero of 99 apps declare x-shc-volumes. Therefore every persistent mount classifies NotRebindable and move selects backup_restore for every deployment, on both estates, today.
  2. MEASURED: MOVE-5: on the compose runtime, every cross-node move is backup_restore, including stateless apps.
  3. MEASURED: 74 of 99 catalogue entries are stateful (211 distinct DATA mounts). Only 5 non-fixture apps are both placeable and stateless; 10 entries run no container at all.
  4. MEASURED: Move wire format is base64-in-JSON, 2 MiB synchronous chunks: 1.33× inflation, ≤65–70 MiB/s ceiling on loopback, round-trip-bound (not CPU-bound, so bigger nodes do not help).
  5. MEASURED: A move stages a full archive on the same disk as the data. No node on either estate has a second disk (610.4 GB sda, ×6). A deployment above ~45% of free disk cannot be relocated at all.
  6. MEASURED: Hetzner volumes: 10 GB–10 TB, single-attach, max 16 per server. A per-mount volume design is infeasible; a per-deployment design caps a node at 16 stateful deployments.
  7. MEASURED: Volume.access_mode CHECK is IN ('rwo','rox'); rwx is refused with X400328VOLACC. No shared filesystem can own a volume row, so both shared-FS classifier arms are unreachable.
  8. MEASURED: DeploymentDir is unconditional and storageapi.Provider.BasePath() has zero callers. Nothing routes a deployment’s bytes onto a storage provider.
  9. MEASURED: 65 of 99 apps declare x-shc-scale, all with max: only; x-shc-scale.<svc>.backup is wired and declared by 0.
  10. MEASURED: Only 7 apps have a full pre_backup/post_backup bracket. 17 more have a pre_backup that is a dump, not a quiesce.
  11. MEASURED: Standby cannot be derived from compose: the four highest-value stateful apps (gitlab, wordpress, nextcloud, matrix) show zero background services and all run background writers.
  12. MEASURED: timeouts.quiesce_hold = 300 s, and overrun escalates to a hard stop.
  13. DERIVED: On a non-shc, non-swarm fabric posture, cross-node integration is unrouted, so co-location is a hard admission constraint, not a score.
  14. MEASURED: dataPhase measures only the deployment directory, so it is blind to swarm named volumes (defect D-1).

Direction: declare x-shc-volumes on a budgeted subset. Reject shared filesystem. Harvest the app-native move path first, because it is free.

Ordered by cost-to-value, with what each buys:

Step 0: free, do it regardless: measure the byte cost

Section titled “Step 0: free, do it regardless: measure the byte cost”

One du -sb sweep over <StateHome>/deployments/*/*/* on all six nodes. It is the same walk MeasureDeploymentTree performs. Without it, every sizing decision below is a guess. This is estate work and is not mine to run.

Step 1: cheapest lever: app-native dump/restore, already declared

Section titled “Step 1: cheapest lever: app-native dump/restore, already declared”

Cost: zero new mechanism. 28 apps already exclude 60 of 211 DATA mounts; gitlab already excludes its entire live tree in favour of a native dump. Extending that pattern to every app whose upstream ships a dump tool shrinks the moved bytes by an order of magnitude for the apps where it works, with no schema change, no cloud spend and no new failure mode.

Step 2: fix the gate before making moves more attractive (defect D-1)

Section titled “Step 2: fix the gate before making moves more attractive (defect D-1)”

dataPhase must measure named volumes as well as the tree, or a swarm move can destroy a source on the strength of a comparison that never saw the data. This is a prerequisite, not a nice-to-have: everything below makes moves more frequent.

Step 3: the storage direction: x-shc-volumes, budgeted

Section titled “Step 3: the storage direction: x-shc-volumes, budgeted”
  • Who declares one: the deployments that are both (a) large enough that the §1.3 byte cost is a real outage, and (b) inside the 16-per-node budget. Start from the Step-0 measurement, not from the catalogue.
  • What it costs: €0.044–0.057/GB/month (R-3.2: confirm on the invoice); ~€4.40/month per 100 GB at the low quote. One compose block per app. Both estates already carry the volume-capable connection.
  • What it buys: the move becomes stop → detach → attach → start, zero bytes on the network, and the staging-disk ceiling disappears for the bound mounts. It also arms volume_failover.go (already built, already fence-proof-gated, already off by default) which is the only automatic node-failure story SHC has.
  • What it constrains: a bound RWO mount is a hard placement pin plus a single-replica clamp. The scheduler must treat it as an input, not a choice.

Step 4: do not build shared storage on this substrate

Section titled “Step 4: do not build shared storage on this substrate”

On Hetzner, CephFS requires block volumes as OSD devices (no node has a second disk), then multiplies them by the replication factor, then adds a Ceph cluster to operate. NFS on a cluster node converts every stateful deployment into a dependency on one machine. Neither buys a property the catalogue asks for: all 65 apps that declare a scale cap declare max: 1, so rwx has no consumer. For the three genuine multi-writer blob cases (nextcloud, immich, peertube), the answer is the S3 plug that already ships.

This is a recommendation about the substrate SHC runs on today, not a permanent doctrine. On a substrate with a managed multi-attach filesystem (EFS, Filestore) the arithmetic changes and this should be revisited.

Step 5: standby as an opt-in optimisation, never a prerequisite

Section titled “Step 5: standby as an opt-in optimisation, never a prerequisite”

Reuse the vocabulary that ships (hooks.pre_backup/post_backup, x-shc-scale.<svc>.backup). Require a self-proving verification per R-3.5; the mariadb job is the model, including its second-connection read-back and its dual-spelling match. An app without one keeps today’s stop-first move, and the selection record must say so by name.


7. Rejected alternatives, with their costs

Section titled “7. Rejected alternatives, with their costs”
alternativewhy rejectedcost if pursued anyway
Shared filesystem (CephFS)Needs OSD block devices; no node has a second disk. Costs block volumes × replication factor, plus a Ceph cluster. Delivers rwx, which 65/99 apps structurally cannot use.~3× the block-volume bill + a new operational surface + the access_mode schema change
Shared filesystem (NFS on a cluster node)Converts every stateful deployment into a single-node dependency, strictly worse availability than todayCheap to build, expensive the first time that node reboots
SHC_STATE_HOME on a shareShares the rqlite data, the vault, the node-local by design move journal and the staging dir. Violates the journal’s premise.~zero to try, catastrophic and hard to reverse
One block volume per DATA mount211 distinct mounts vs a 16-per-server cap. greenbone alone declares 18.Infeasible; fails at the first dense node
Derive standby from the compose graphMEASURED to fail on gitlab, wordpress, nextcloud, matrix, the four apps where being wrong costs the mostA confident wrong answer, which is the worst outcome available
Start-first cutover without standby65/99 apps are max: 1; two writers on one dataset. Today’s stop-first is correct.Data loss, with a green health probe (the exact 2026-08-09 shape)
Raise cloneRecoverChunkBytesThe transfer is round-trip-bound, not chunk-bound, and >3 MiB raw exceeds internal_rpc’s 4 MiB body capEvery chunk starts failing, the cliff the streaming replaced

  • No estate contact, so: no per-deployment byte sizes (I have exactly two: nextcloud 8.44 GB, openobserve ≈5 GB), no read of either estate’s fabric posture row, no confirmation of whether the integration table is still empty, no live cross-node ship measurement.
  • The ship benchmark is a loopback ceiling, not a field measurement. Real cross-node throughput is bounded above by 65–70 MiB/s and I do not know how far below.
  • Hetzner volume pricing is contradictory across third-party sources (€0.044 vs €0.057/GB/month). Not resolved; do not put either in a plan without an invoice.
  • The x-shc-volumes declaration effort is unestimated. I know it is one compose block per app; I do not know how many of the 74 need a size chosen by someone who knows the workload.

#questionwhy it blocks
R-3.1Are block volumes a budgeted per-deployment escape hatch (16/node) or something the catalogue declares broadly?Decides whether §6 Step 3 is a shortlist or a campaign
R-3.2Confirm the Hetzner volume price on an actual invoice€0.044 vs €0.057 is a 30% swing on every number in §2.1
R-3.3Must a non-shc/non-swarm fabric posture make co-location a hard admission constraint?Without it a scheduler can produce a deployment that cannot reach its database
R-3.4For a consistency group: co-locate (latency) or spread (failure domain)?Directly contradictory scoring terms; not derivable from the code
R-3.5Must a standby declaration carry its own fresh-connection verification?The mariadb precedent says an unverified declaration is worse than none
R-3.6Is defect D-1 (data gate blind on swarm) a P1 to fix before any rebalancer ships?Everything in this lane makes moves more frequent

The catalogue censuses in §1.2, §4.2 and §4.3 are single-pass reads of apps/*/{app.yaml,compose.yaml,backup.yaml} at shc 80e0c4441. Each is restated here so the number can be re-derived without the scratch scripts.

Mount-type census / stateful count (§1.2). For every service’s volumes: entry, split source:target[:mode]; then apply DetectMountType verbatim: a source starting with temp/tmp is RUNTIME; a source starting with / is SYSTEM (the source decides; the container-side target carries no authority); a /tmp target on a relative source is RUNTIME; ./ or ../ is FILE; a bare name that exists as a path inside the app dir is FILE; anything else is DATA, i.e. persistent. Long-form entries with type: tmpfs are ephemeral. Expect 362 DATA references / 211 distinct, 74 of 99 entries stateful.

Placeable-and-stateless count (§1.2). Of the 25 with no DATA mount, subtract the entries whose services: map is empty (grep -c the nine apps/standalone/external-* plus apps/standalone/s3, all declare zero services) and the eight under apps/fixtures/ and */tests/apps/.

Hook census (§4.3). hooks: is in app.yaml, not backup.yaml;the job bodies live in sibling *.job.yaml files. Counting jobs: keys in backup.yaml gives the wrong answer (it finds one pre_backup, in ciso-assistant); counting hooks: keys in app.yaml gives 24 / 7.

Scale census (§2.2, §4.3). grep -l x-shc-scale apps/*/*/compose.yaml → 65; every one matches max:\s*1; parsing each service’s x-shc-scale map for a backup key → 0.

Ship-leg benchmark (§1.3). ~120 lines of Go: write an N-MiB file to tmpfs; read it in 2 MiB chunks (cloneRecoverChunkBytes); for each chunk build {"args":[],"kwargs":{"session":…,"offset":…,"chunk_b64":<base64>}}, json.Marshal, POST to a loopback net/http handler that json.Unmarshals, base64-decodes and WriteAts. Time the loop. This is ce/daemon/services_clone_remote.go::shipBakToPeer with the network and TLS removed, so the result is an upper bound. Expect 1.33× wire inflation exactly and 60–70 MiB/s, roughly unchanged at GOMAXPROCS=2.