The deployment binding model (proposal)
Status: open design. Not ruled and not built. The Swarm cell it describes went with the new line (SHC-1092), and nobody has ruled on whether the bound and unbound predicate still applies. Written 2026-08-02, from three parallel read-only audits of ce’s volume, placement and enforcement surfaces, re-verified against ce and shc at
2815168(shc pins ce atv0.0.12-0.20260731190259-5da1b0869c7a). No production estate was contacted.
The owner’s framing, which this document makes precise:
“Let’s not have ‘this deployment is on this node’, but ‘this deployment has critical state on this node’ or ‘this deployment is bound to this node’. Unless it’s moved or becomes unbound via unpinning, that node cannot be removed. However where there are other potentially available nodes, or you could add another node, it would simply start working there, of course only in a swarm environment. I don’t want to ever be in a scenario where you can lose state.”
Two sharpenings carried throughout:
- Bound-ness is a property of the DEPLOYMENT, not of current topology. An unbound deployment on a one-node cluster is still unbound. Removing that node makes it UNSCHEDULABLE (availability), not LOST (data). Those deserve different treatment: losing state is a hard refusal, losing availability is a warning an operator may accept.
- Swarm does not make a deployment safe; shared storage does. A swarm service with a critical node-local volume is the most dangerous configuration in the system, because swarm reschedules the task and starts it against an empty volume.
This document consumes, and must not contradict, the node addressing model. Its slice 11 (“destroy refuses, drain relocates”) is the mechanism at the terraform boundary; this document supplies the predicate that refusal should test. Its Q11 is refined here rather than re-asked. For the runtime-neutral cross-node data plane this model must eventually express, see network-fabric.md.
A word this project has already spent three times
Section titled “A word this project has already spent three times”Before anything else, because the design is unreadable otherwise. “Bind” has three unrelated meanings in SHC today, and one of them means close to the opposite of the owner’s:
| spelling | today’s meaning | scope |
|---|---|---|
bind | where the daemon LISTENS: bind_ip, internal_api.bind, SHC_RQLITE_BIND_ADDR | a node’s own socket |
binding | a compose mount that has an x-shc-volumes block-volume row: volume.ParseVolumeBindings, “a BOUND swarm DATA mount” (ce/modules/app/install_runner.go:3089) | a mount |
bound | the owner’s: a deployment whose critical data can be served only by named nodes | a deployment |
The second is the dangerous one. In today’s code a “bound” mount is a block volume, which is one of the LEAST node-bound things in the system, because a location-scoped block volume detaches and reattaches to any node in its location. A reader who has internalised the code’s spelling will read the owner’s sentence backwards.
Proposal, and it is cheap: the code sense is renamed.
x-shc-volumes mounts become provisioned (they carry a volume row)
versus plain; hasBinding becomes hasVolume. bind keeps its
addressing-model meaning unchanged (that glossary is authoritative and
this document does not touch it). bound / unbound / bound_to are
reserved for this model, at deployment scope only.
Problem
Section titled “Problem”SHC cannot answer “would removing this node lose data”. It can answer “how many deployment rows point at this node”, and it treats that as the same question. It is not.
Three facts, each verified:
1. The gates count deployments, not state. shc cluster rm refuses on
X400017NODDEP: “node has stacks, rerun with force=true to override”
(shc/modules/cluster/admin_router.go:171-183), a pure COUNT over
deployment.node_id. It cannot tell a stateless nginx from the only copy
of a database, and force=true bypasses it entirely, after which
DeploymentRepo.DeleteByNode (ce/modules/app/deployment_repo.go:162-180)
sweeps the rows so the ON DELETE RESTRICT foreign key stops objecting.
The bytes stay orphaned on a disk nothing points at any more.
shc node forget has no deployment gate at all; its handler calls
Remove(ctx, m.ID, true) with force hardcoded
(shc/modules/cluster/lifecycle_router.go:305), and that is the verb the
terraform provider’s Delete invokes.
2. The predicate already exists, correct, and is reachable from one
verb. ClassifyMountDecision
(ce/daemon/clone_move_adapters.go:213-266) reduces per-mount facts to
Ephemeral | Rebindable | NotRebindable | CrossProvider, keying the block
case on a topology predicate (a non-empty persisted location) rather
than a vendor name. ClassifyDeadMountDecision
(ce/daemon/clone_dead_move_adapters.go:353-379) goes further and names
the owner’s condition outright: DeadMountLost is “a local-only
persistent mount; the data lives ONLY on the dead node’s disk and does
NOT survive. Refused unless the operator passes --discard-local”
(ce/modules/clone/move_presumed_dead.go:73-77), enforced by
X400619CLNDLM. That is the owner’s model, already implemented, already
fail-closed, already with an explicit override, and it is reachable only
from shc move --presumed-dead. Install does not consult it. Node
removal does not consult it. cluster repair does not consult it.
3. The one configuration that loses state silently is the default.
Detailed in the decision table. In summary:
ApplyDeployPlacement returns the compose unchanged when no node was
requested (ce/modules/app/swarm_placement.go:33-36), and an unbound
swarm DATA mount is deliberately kept as a docker-managed named volume
that “swarm provisions per-node on whichever node the task lands”
(ce/modules/app/install_runner.go:3066-3090). So shc install <db-app> --runtime swarm yields critical state in a per-node volume with no
placement constraint. Twelve lines below that branch, the same file
refuses the equivalent configuration for a provisioned mount:
X400324VOLSWMNB, because it “would shadow the attached device with an
empty volume (medium #18 data-loss case), and on swarm there is no
single-node guarantee”. The unbound case does exactly what the bound case
is forbidden from doing.
And the guard an operator would expect to catch this does not exist.
shc/docs/reference/cli/storage.md:233 states “Swarm-mode stacks require
a shared provider (nfs or cephfs); local is rejected for Swarm”, and
the architecture doc names the error code X400044SWRLCL and the function
ResolveDefaultStorage. Grep across ce and shc: the code, the function
and the error code do not exist. They are a description of Python-era
behaviour that was never ported.
The model
Section titled “The model”Two independent per-mount properties. One derived per-deployment verdict. Nothing stored.
Durability: would losing these bytes be data loss?
Section titled “Durability: would losing these bytes be data loss?”durability(m) ∈ { critical, ephemeral }Resolved by a fixed three-step precedence, and every step is reportable:
- Declared. The app’s
backup.yamlgains adurability:block naming mounts explicitly. Author intent wins. - Inferred, structurally:
ephemeralwhen the mount is tmpfs, an anonymous volume, a socket / device node / FIFO, a read-only bind, under a pseudo-root (/proc /sys /dev /run /var/run /tmp), aRUNTIMEmount, or aFILEmount re-derived from the app source on every render. Every one of these judgements is already written and already exercised against 73 apps:ce/modules/backup/hostbind.go:100,:210,:222,:269-281,:382-385andce/core/volume/resolve.go:93. - Otherwise
critical. Fail-closed. Unknown binds.
Anchor: which nodes can serve these bytes without copying them?
Section titled “Anchor: which nodes can serve these bytes without copying them?”anchor(m) ⊆ Nodes, or UNKNOWN| mount shape | anchor |
|---|---|
compose DATA mount (host bind under <deployDir>/mounts/<name>) | { the node holding the deployment dir }, a singleton |
| absolute host bind | { that node }, a singleton |
| swarm docker-managed named volume, service PINNED | { the pinned node } |
| swarm docker-managed named volume, service UNPINNED | UNKNOWN |
| RWX shared filesystem (nfs / cephfs), verified mounted on N | N |
location-scoped block volume (volume.location != "") | { n : n.location == volume.location } |
| block volume with no location stamp | { current attachment node } |
Note the asymmetry that makes anchor the right primitive rather than a
shared/node-local label: a location-scoped block volume in a
single-node location is a singleton anchor and binds, while the same
provider in a two-node location does not. The topology decides, and
ClassifyMountDecision already computes exactly this reduction.
The binding predicate, stated exactly
Section titled “The binding predicate, stated exactly”A deployment
Dis BOUND to nodenif and only ifDhas a mountmwheredurability(m) == criticalandanchor(m) == { n }.
critical(D) = { m ∈ mounts(D) : durability(m) == critical }bound_to(D) = { n : ∃ m ∈ critical(D) with anchor(m) = {n} }unanchored(D) ⟺ ∃ m ∈ critical(D) with anchor(m) = UNKNOWNunbound(D) ⟺ bound_to(D) = ∅ and not unanchored(D)bound_to is a set, which is the owner’s plural and it is load-bearing:
a deployment with critical local state on two nodes (after a partial move,
or with two mounts whose singleton anchors differ) is bound to both, and
neither may be removed. A block-volume deployment in a two-node location
is bound to neither, because neither singleton condition holds.
What is NOT a binding factor, and why the owner’s list needs trimming
Section titled “What is NOT a binding factor, and why the owner’s list needs trimming”The owner’s candidates were: explicit pin; a runtime that cannot reschedule (compose, podman); any critical volume on a node-local provider; host paths and bind mounts. Two of those four are not data bindings, and conflating them is what makes the removal gate either unsound or useless.
An explicit --node pin does not bind. It is a placement declaration.
If every critical mount of a pinned deployment sits on a shared filesystem,
removing the pinned node loses availability, not bytes; clear the pin, or
shc move, and it runs elsewhere with the same data. A pin is nonetheless
load-bearing in the other direction: a pin is what makes an otherwise
dangerous configuration safe, because it turns anchor = UNKNOWN into
anchor = { pinned node }. Pins are the cure, not the disease.
“Compose and podman cannot reschedule” does not bind either. It is a
scheduling fact, and it is a consequence of how compose renders, not an
independent cause. What actually binds a compose deployment is that
ResolveVolume rewrites every DATA mount to a host bind under
<StateHome>/deployments/<t>/<s>/<e>/mounts/<name>
(ce/core/volume/resolve.go:148-155, ce/core/paths/paths.go:712) on the
node that ran the install, so its critical bytes are on that node’s disk.
A compose deployment with no critical mounts (a stateless proxy) is
unbound, and the fact that compose will not move it by itself makes it
UNSCHEDULABLE elsewhere until an operator acts. shc move resolves that in
seconds with no data copy. That is a warning, not a refusal.
Trimming these two is not pedantry. If “pinned” and “compose” both counted as bindings, then on both live estates every deployment would be hard-unremovable, the refusal would fire constantly for reasons that are not data loss, and operators would learn to reach for the override. A refusal that fires when nothing is at stake is how a refusal stops being believed.
A binding factor the owner did not list, and it is the sharpest one.
anchor(m) = UNKNOWN. A swarm service with an unpinned docker-managed
named volume has its bytes on exactly one node’s disk; SHC simply cannot
say which. SwarmServiceStates shells docker stack services --format {{json .}} and decodes {Name, Replicas, Image}
(ce/modules/app/swarm_driver.go:82-94); there is no node field, and
nothing writes observed placement back. Meanwhile stampNode records the
node that handled the install request
(ce/modules/app/store.go:160-176) and never updates it. So the deployment
row asserts a location the scheduler is free to ignore, and after one
reschedule the row is simply wrong.
This deserves its own state name because it is not a kind of bound-ness; it is the absence of the fact bound-ness is computed from.
Three states
Section titled “Three states”| state | meaning | removing a node in scope |
|---|---|---|
| UNBOUND | every critical mount has an anchor larger than one node, or there are no critical mounts | availability question: WARN |
| BOUND | some critical mount’s anchor is a known singleton; bound_to names those nodes | data question: REFUSE |
| UNANCHORED | some critical mount’s anchor is UNKNOWN | not a design state; a defect state |
UNANCHORED is not a third posture an operator may choose. It is the configuration in which the question “would this removal lose data” has no answer, and the whole of § enforcement turns on making it unrepresentable.
Critical versus temporal: one property, three sources
Section titled “Critical versus temporal: one property, three sources”The owner asked for “a concept of critical volumes versus temporal volumes; only the critical ones bind”. Resolved as one property with three sources, not two properties, and deliberately not by reusing the backup signal as-is.
Why backup-inclusion is the wrong signal to reuse directly
Section titled “Why backup-inclusion is the wrong signal to reuse directly”The correlation is high and it is causal: the backup layer exists to
preserve exactly the bytes whose loss is unacceptable, and 23 of 73 apps
already reason about it in prose, well. apps/highlight/backup.yaml
excludes kafka-data / zookeeper-data / redis-data as “pure
ingest-queue + cache state, NOT durable user data”; apps/huly/backup.yaml
keeps elastic despite it being derivable because “rebuilding is
O(every document), not O(delta)”. Those are criticality arguments, in the
right place, by the right person.
Three reasons they still cannot be the binding signal:
It inverts on the most critical mounts. ShouldBackupMount returns
false when a converter owns the mount
(ce/modules/backup/archive.go:1302-1310); the database dump replaces the
raw files. A naive “backed up implies critical” therefore classifies a
Postgres data directory as NOT critical. That is the single worst
misclassification available.
It is a per-run runtime fact, not a static property. Backup inclusion
depends on which capture path ran, whether a converter fired, and whether
--path narrowed the snapshot. Per sharpening 1, bound-ness must be a
property of the deployment. A property that changes because a backup ran
with different flags cannot gate a node removal.
exclude.mounts is an author preference with a different meaning. An
author who excludes a 100k-file feed to keep snapshots small has said
“don’t snapshot this”. They have not said “you may destroy the node holding
it”. Overloading one key with a second, destructive meaning is precisely
the trap ce/modules/backup/types.go:709-729 already refuses elsewhere,
where include.files / exclude.files were deleted because one key could
only reach one of two capture paths.
The resolution
Section titled “The resolution”durability is declared in backup.yaml in a separate durability:
block, alongside exclude.mounts and never derived from it. That keeps
the reasoning where the reasoning already lives, and keeps the two
questions apart:
exclude: mounts: [vt-data] # too big to snapshot; a re-feed is cheaperdurability: critical: [vt-data] # ...but losing the node still costs a re-feedexclude: mounts: [kafka-data, zookeeper-data, redis-data]durability: ephemeral: [kafka-data, zookeeper-data, redis-data] # genuinely disposableephemeral means “I accept losing this without being warned”. That is
the whole contract, and it is why the middle case (expensive to rebuild
but not truly lost) classifies critical by default. An author who wants
the rebuild to be silent says so explicitly.
Two cross-checks fall out for free and should be reported by
shc app lint: a mount declared ephemeral but not excluded from backup
is a smell (you are snapshotting something you call disposable), and a
mount excluded from backup with no durability declaration is where an
author most likely meant one.
The classifier is promoted, not duplicated
Section titled “The classifier is promoted, not duplicated”The judgements above are made in six independent implementations today,
each re-parsing the compose with its own notion of a source:
DetectMountType (ce/core/volume/resolve.go:93), backup’s
DiscoverHostBindSources + ExcludeHostBindMode
(ce/modules/backup/hostbind.go), ShouldBackupMount
(ce/modules/backup/archive.go:1302), DiscoverUnboundNamedVolumes
(ce/modules/backup/namedvolume.go), PersistentMountsFromCompose
(ce/daemon/clone_move_adapters.go:551), and DeployedStackVolumes
(ce/modules/app/volumes_deployed.go:47). They agree by code review, not by
construction, and they already disagree in one concrete place: an absolute
host bind is CRITICAL to backup (“host-bind data must MOVE with the app,
not merely be warned about”, hostbind.go:23-31) and EPHEMERAL to move
(“classified ephemeral; SHC data never lives there by convention”,
clone_move_adapters.go:551).
They collapse into one classifier in core/volume, with backup, move,
dead-move, the HUD and the new binding predicate as consumers. Collapsing
six parsers to one is worth doing on its own; it is a precondition here.
Visibility, and failing safe
Section titled “Visibility, and failing safe”An operator must be able to see a classification before it refuses something, and the refusal must be reproducible as a query. Two verbs:
shc <stack> binding: per mount: name, kind,durabilityand which of the three sources decided it,anchor(the node set), and a one-line reason. Then the rollup: state, andbound_to.shc node bindings <node>: the inverse. Every deployment bound to this node, with the mounts that bind it named. Runnable before attempting a removal, not discovered from a refusal message.
Both render the same reasons the refusal renders, from the same function.
The nearest thing today is shc node info <name>, which joins the node row
with shc.node.deployments and returns {stack, env, status, runtime}
(ce/modules/node/deployments_router.go:26-31): placement, no binding, no
reason.
Misclassification fails safe in both directions. Unknown durability is
critical; unknown anchor is UNANCHORED, which refuses. The cost of
over-binding is one shc move an operator did not need. The cost of
under-binding is a restore, or nothing to restore from.
The decision table
Section titled “The decision table”The owner’s question: with swarm, do you have to move or not?
You have to move under exactly the same condition as compose: when a critical mount’s anchor is a singleton. Swarm changes nothing about whether you must move. It changes what happens if you don’t. On compose, not moving means the deployment stops, loud, recoverable, the bytes are still there. On swarm, not moving means the task restarts on another node against an empty volume, silent, and the bytes are stranded on a disk the running app is no longer looking at.
So swarm is strictly more dangerous than compose for the identical configuration, which inverts the intuition that swarm is the highly available runtime. Swarm buys availability only for deployments that were already unbound.
| # | runtime | pin | critical mount’s storage | what happens when that node goes away | must move? | state |
|---|---|---|---|---|---|---|
| 1 | compose / podman | implicit | node-local disk | deployment stops; row still points at the dead node | YES | BOUND |
| 2 | compose / podman | implicit | shared filesystem | unreachable today; see below | n/a | would be UNBOUND |
| 3 | compose / podman | implicit | location-scoped block volume | volume survives detached; move fences and rebinds | no | UNBOUND in-location |
| 4 | compose / podman | implicit | none critical | deployment stops | no | UNBOUND, unschedulable |
| 5 | swarm | --node | node-local named volume | task goes Pending: no suitable node, LOUD, bytes intact | YES | BOUND |
| 6 | swarm | none | node-local named volume | task reschedules; docker creates a fresh EMPTY volume; the app comes up blank | YES | ⚠ UNANCHORED |
| 7 | swarm | either | shared filesystem | unreachable today | n/a | would be UNBOUND |
| 8 | swarm | none | provisioned RWO block volume | ApplyVolumePlacement pins to the attachment node and clamps replicas: 1; fenced rebind on move | no | UNBOUND in-location |
| 9 | swarm | none | none critical | reschedules cleanly | no | UNBOUND |
Every cell is critical durability except 4 and 9; the ephemeral variant
of any row is UNBOUND by definition and is not worth nine more rows.
Cell 6 is the one that loses state silently, and it is today’s default
Section titled “Cell 6 is the one that loses state silently, and it is today’s default”Reachable by shc install <any stateful app> --runtime swarm with no other
flag. Nothing refuses it, nothing warns, and shc ls reports a node.
Cell 8 (the safe swarm shape) requires the app to declare
x-shc-volumes, and grep over shc/apps/ returns x-shc-volumes zero
times across all 73 catalogue apps (x-shc-scale appears in 63 of them,
so the extension mechanism is well understood; this particular one is
simply unused). So every stateful swarm install of every shipped app lands
in cell 6.
Two further hazards on the same shape:
shc scale <stack> <svc> Nhas no volume guard at all (ce/modules/app/service.go:859-897). N replicas against one node-local volume name is N forked datasets. Thereplicas: 1clamp exists only for provisioned RWO (ce/modules/app/swarm_placement.go:104-161).- FILE mounts fail the opposite way. SHC never converts a swarm DATA
mount to a host bind, so the loud “bind source does not exist → task
Rejected” failure does not apply to DATA. It applies to FILE, and that is
an INFO log (
X200995PMTFIL). DATA fails silently-empty; FILE fails loudly-unschedulable. Exactly inverted from what safety wants.
Cells 2 and 7 are unreachable, and that is the second finding
Section titled “Cells 2 and 7 are unreachable, and that is the second finding”The owner’s model requires an escape hatch: “become unbound” by moving critical data onto a shared provider. SHC cannot execute it.
DeploymentDir is unconditionally <StateHome>/deployments/<t>/<s>/<e>
(ce/core/paths/paths.go:712) and MountsDir unconditionally
<deployDir>/mounts (ce/modules/app/install_runner.go:3023). The storage
provider is never consulted on that path. storageapi.Provider.BasePath()
and CreateDeploymentFolder (the interface that would place a
deployment’s data on a provider) have zero production callers in
either repo. NFS and CephFS providers are real: they are registered, linked
to nodes, and genuinely mounted at /mnt/shc/nfs and /mnt/shc/cephfs.
Nothing routes a deployment’s mounts onto them.
--storage <name> is therefore an unvalidated label. Nothing checks
that the provider exists, that the target node is linked to it, or that it
is legal (ce/modules/app/service.go:475-478 stores it verbatim;
install_refusal.go has no storage gate). It changes no placement. And it
reaches exactly two consumers, one of which is dangerous:
The move classifier trusts it. ClassifyDeployment falls back to
deployment.storage as the provider for any mount without a volume row,
which is every mount in every shipped app
(ce/daemon/clone_move_adapters.go:117-120, defaulting to "local" when
empty). ClassifyMountDecision then returns Rebindable for nfs /
cephfs whenever the target node carries a store_node link
row, a row AttachAllNodes writes even when the node-side client mount
failed, warned as X503085SHFSMT. runRebindMove then skips the rebind
entirely, on the reasoning that “a rebindable mount without a volume id is
an RWX shared FS already visible on the target; nothing to detach/attach”
(ce/modules/clone/move_strategy.go:429-445), re-renders on the target and
starts.
So shc install nextcloud --storage nfs (the storage documentation’s own
worked example, shc/docs/reference/cli/storage.md:229) records "nfs",
places the bytes on local disk anyway, and arms a later shc move to
relocate the deployment onto a node where those bytes are not. The only
thing on that path is a warning.
The same unverified claim is trusted by the dead-node classifier
(ce/daemon/clone_dead_move_adapters.go:205-208), where DeadMountShared
means “the data lives on the share”, on the path the code itself calls
“the most dangerous path in the product”.
Where enforcement lives
Section titled “Where enforcement lives”Three points, and the ordering between them is the argument.
1. Install: make UNANCHORED unrepresentable, do not refuse the workload
Section titled “1. Install: make UNANCHORED unrepresentable, do not refuse the workload”The install-time guard is what makes the removal-time guard usable. This is the whole case for enforcing at install, and it is structural rather than a matter of taste.
The removal guard must be fail-closed: if SHC cannot prove a removal is
safe, it refuses. But anchor = UNKNOWN on an unpinned swarm service means
the bytes might be on any node, so a sound fail-closed removal guard must
refuse every node removal for as long as one UNANCHORED deployment
exists anywhere in the cluster. That leaves exactly two options, and both
are bad: allow the removal anyway (unsound: it loses data), or block all
removals (useless: the cluster becomes unrotatable, which directly
contradicts the addressing model’s rotation goal).
Making UNANCHORED unrepresentable at install is the only way out. It is
also the only enforcement point that helps an operator who never runs
node rm at all, including the case in § when a bound node
dies, which is the
case that most directly threatens the owner’s bar.
The mechanism is a pin, not a refusal. At render, a swarm deployment
with any critical mount whose anchor would be a singleton is stamped with
node.hostname==<holder> (whether or not --node was passed), reusing the
mechanism ApplyVolumePlacement already implements for provisioned RWO,
including its replicas: 1 clamp. Cell 6 becomes cell 5. Nothing is
refused, no catalogue app breaks, and no operator action is required.
Why pin rather than refuse: refusing cell 6 makes stateful swarm installs
impossible for all 73 catalogue apps, since cell 8 needs x-shc-volumes
that none of them declare and cell 7 does not work. That is a product
regression with no safety gain over pinning; the pin is exactly the fact
the deployment already implies, made explicit.
Refusal is reserved for requests with no safe interpretation:
| request | verdict |
|---|---|
| swarm + critical local mount, no pin | auto-pin (derived), report it |
explicit --unbound / --storage <shared> that cannot be honoured | refuse: the ask is unsatisfiable, and today it is silently accepted |
shc scale <stack> <svc> N, N > 1, on a BOUND deployment | refuse: N replicas, N forked datasets |
| clearing a derived pin while the mount is still node-local | refuse: that manufactures UNANCHORED on purpose |
| clearing an operator pin on an UNBOUND deployment | allow: that is what unpin is for |
The derived pin must be distinguishable from an operator’s --node, so
that shc unpin can clear the second and refuse the first. That is one
boolean alongside the placement, and without it “unpin” has no coherent
definition.
One honest cost of auto-pinning, stated plainly: it makes swarm’s
non-availability for stateful workloads silent unless bound-ness is
visible. That is the entire justification for
the two inspection verbs being part of the
same slice, not a later nicety. A pinned deployment must read BOUND in
shc ls, or SHC has quietly replaced a data-loss bug with an availability
lie.
2. Node removal: split DATA LOSS from AVAILABILITY
Section titled “2. Node removal: split DATA LOSS from AVAILABILITY”Today --force collapses both into one bypass. Splitting them is the
change that makes “I don’t want to ever be in a scenario where you can
lose state” enforceable rather than aspirational.
| condition | verdict | escape |
|---|---|---|
n ∈ bound_to(D) for some D | HARD REFUSAL: not --force-able | shc move D --node <other> (relocates the bytes), or an explicit shc D abandon --discard-local naming the mounts being given up |
any D is UNANCHORED | REFUSE, fail-closed: the message names the uncertainty, not a deployment | make it anchored (pin or move), then retry |
D unbound, n was its last runnable node | WARN: availability, not data | plain --force |
| otherwise | allow | none |
The escape verb is not new machinery: --discard-local is the existing
precedent from move_presumed_dead.go, and it reads right: the operator
names what they are abandoning rather than passing a generic override.
Non---force-able matches EnsureVoterRemovable’s posture, which is
already the house pattern for “this guard exists to stop you wedging
something you cannot un-wedge”
(ce/modules/cluster/service.go:1715-1810).
Four removal paths must all consult this, and today they behave four different ways:
| path | today | under this model |
|---|---|---|
shc cluster rm <node> | X400017NODDEP counts deployments; --force bypasses | split verdict; the data leg is not bypassable |
shc node forget <node> | force hardcoded, no deployment gate at all (lifecycle_router.go:305) | keeps force for a node that is already gone, but reports what it is orphaning |
shc cluster repair | evicts any node unreachable or unseen 24h, force hardcoded | must refuse for a node in any bound_to; may sweep the rest |
terraform destroy of shc_node | calls forget, so no gate at all | the addressing model’s slice 11 routes it to the gated removal; this model supplies the predicate |
shc cluster repair’s 24-hour sweep is the sharpest instance of the
owner’s concern in the current code: a day-long partition silently evicts
every deployment row on a node, plus its node-local backup rows.
3. Continuously: derive, never store
Section titled “3. Continuously: derive, never store”Both the transitions and the temptation are real. Bound-ness can change under a running deployment at least four ways:
shc storage nfs disabledeletes the provider row through the raw repository path, deliberately bypassingDeleteProvider’s in-use guard (ce/modules/storage/shared_manager.go:245-249, comment: “Bypass the IsManaged guard… this orchestrator is the only legal entry point”). Every deployment naming that provider silently rebinds. The guard it drives around already implements the right join (ce/modules/storage/service.go:388-420).shc storage unlink <provider> <node>: a shared anchor shrinks; nothing re-classifies.shc <stack> update: a catalogue bump can introduce a new DATA mount, and update re-renders while dropping--node(ce/modules/app/commands/stack_lifecycle.go:691-693). An UNBOUND deployment silently becomes BOUND, or on swarm UNANCHORED.- A swarm reschedule: the holder changes; the row does not.
The tempting fix is a bound column plus a reconcile cron. Do not. A
stored bound-ness flag is exactly the class of thing that goes stale and
turns a guard green, the failure mode the occupancy guard’s own
fail-closed comment is written against. Bound-ness is derived fresh at
every decision point from four inputs that already exist: the rendered
compose’s mount set, volume rows plus attachment, storage-provider link
rows, and placement constraints parsed from the rendered compose. Only the
rendered answer is cached, for display; no guard may read the cache.
What is continuous is small and cheap: the two operator commands above
(1 and 2) must run the in-use join before performing a rebinding
transition, and update (3) must re-derive and re-pin rather than only
re-render. Those are three call sites, not a cron.
When a bound node dies, and no gate can refuse that
Section titled “When a bound node dies, and no gate can refuse that”This is the case that most directly threatens the owner’s bar, and the one no refusal reaches. Five behaviours, in priority order.
1. Never silently reschedule, and this is already paid for at install
Section titled “1. Never silently reschedule, and this is already paid for at install”For swarm, a BOUND deployment carries a placement constraint, so swarm’s
own scheduler refuses to place it elsewhere. The task goes
Pending: no suitable node. That is loud, correct, and non-destructive:
the bytes are intact on a disk that is merely unreachable.
This is the connective tissue between the two sections and the strongest argument for enforcing at install: the install-time pin IS the dead-node guard. Everything below is about surfacing and recovery; this is the only part that prevents the loss, and it must be in place before the node dies, because nothing can be added afterwards.
2. Surface it as STRANDED, and name the node
Section titled “2. Surface it as STRANDED, and name the node”Today node_poller stamps the node unreachable and its containers
unknown, while the deployment row still reports an ordinary placement.
There should be a first-class deployment state meaning “this deployment’s
critical data is on a node that is not answering”: shc ls shows it, the
HUD shows it, and it does not auto-resolve.
Distinguish it from ordinary unhealthiness, because the operator response is different: a stranded deployment must not be restarted, redeployed or scaled, and every one of those verbs should say so.
3. Wait, by default
Section titled “3. Wait, by default”An unreachable node is not a dead node. The default posture is to wait,
which is what SHC does today for deployments, and explicitly not what
shc cluster repair does for node rows. Repair may sweep an unreachable
node carrying no bound deployments; it must refuse one that carries them,
and say which.
4. One explicit escape, which names the loss
Section titled “4. One explicit escape, which names the loss”shc move <stack> --presumed-dead already is this verb, and it already
behaves correctly: it classifies every mount, refuses X400619CLNDLM when
any is local-only, treats an unreadable app template as lost rather than
assuming safety (MountsProven), and requires --discard-local to
proceed. The model’s contribution is not to build it but to make its
classifier THE classifier, so the same reasoning is reachable from every
path instead of one verb.
5. Restore from backup is the recovery, and it must be honest about what it cannot replay
Section titled “5. Restore from backup is the recovery, and it must be honest about what it cannot replay”--discard-local currently means “abandon the local mounts”. Whether that
costs four hours or everything depends on facts the operator cannot see:
whether a backup ever captured that mount, how old it is, and whether a
restore would actually replay it. Three real ways the answer is “no”:
- a converter ran with no matching
jobs.restoreentry;apps/redis/backup.yamldocuments this trap at length, and the restore “reports success and comes up with an empty keyspace”; - a host bind excluded by backup policy;
- a swarm named volume from before
ce/modules/backup/namedvolume.golanded, when a.bakof such a deployment shipped with no volume data at all.
Proposal: the binding model carries a recoverability fact per critical
mount: last capture time and whether a restore path exists; and
--discard-local prints it per mount before proceeding. That is the
difference between an operator accepting a four-hour gap and an operator
accepting total loss, and today the two are indistinguishable at the
prompt.
This is the honest answer to “no gate can refuse a node dying”: the system cannot prevent it, so it must make the cost of every available response legible before the operator commits to one.
Migration
Section titled “Migration”Both estates (promanager/hel1 and bitspur/hel1) are three nodes: one
compose-runtime node and two swarm-runtime nodes, per
promanager-ops/hetzner/hel1/infra/main.tf:139-140 and
bitspur-ops/hetzner/hel1/infra/main.tf:144-145. Both configure
storage.volumes.provider as hcloud, which is the block-volume backend
and never fires because no catalogue app declares x-shc-volumes.
Can this be adopted without recreating anything? Yes
Section titled “Can this be adopted without recreating anything? Yes”The model is pure derivation over facts already on disk. It adds no schema, moves no bytes, and re-renders nothing by itself. Adoption in waves 0 through 2 (see slices) touches zero deployments.
The behaviour changes are all in the refusal direction, on paths the estates can hit but do not exercise routinely:
shc cluster rm/node forget/terraform destroyof a node holding bound deployments begins refusing where it previously evicted and orphaned. This is the same path the addressing model’s slice 11 already proposes to gate; the two land together or slice 11 lands first with a count-based predicate and this model refines it.shc cluster repair’s 24-hour auto-eviction begins refusing for such a node.
The one wave that changes a render is the swarm auto-pin. It re-renders swarm deployments to add a placement constraint, and neither estate has any, so on these two estates it is a no-op. See below.
What is bound today on the two estates
Section titled “What is bound today on the two estates”Derived from the ops repositories and the code, without contacting either estate.
bitspur-ops/hetzner/hel1/deployments/corp.tf declares eight
shc_deployment resources: postgres (as affine-db), affine,
postgres (as huly-db), minio, huly, postgres (as keygen-db),
keygen, and license-broker. None of them sets node, runtime or
storage; all three attributes exist on the resource
(terraform/provider/internal/provider/deployment_resource.go:205,217,301)
and all three are absent from every block. promanager-ops’s deployments
root declares no apps by design; the promanager application deploys from
its own repository’s CI, which was not inspected.
Three code facts turn that into a classification:
- The deployment runtime defaults to compose, not to the node’s
runtime:
Runtime: RuntimeComposein the model literal (ce/modules/app/service.go:444), overridden only whenreq.Runtime != ""(:487-489). So every one of the eight is a compose deployment, including any that landed on a swarm-runtime node. - Compose DATA mounts become host binds under the deployment dir on the node that ran the install. Every one of these apps has critical DATA mounts: three Postgres instances, a MinIO object store, plus each application’s own state.
deployment.storageis NULL, so the move classifier defaults the provider to"local"(clone_move_adapters.go:117-120), which is the correct answer here, arrived at by luck rather than by validation.
Verdict: on day one, every deployment on both estates classifies BOUND, each to exactly one node. Nothing is reclassified into a more dangerous state; nothing becomes newly unsafe. What the model adds is that the fact is now stated, and that removing the holder node stops being possible by accident.
Cell 6 is not currently occupied on either estate. No estate deployment
sets --runtime swarm, so none is UNANCHORED. The trap is the product
default for anyone who types --runtime swarm; it is not a live exposure
on these two clusters. That distinction should inform urgency: this is
urgent for the product, not for the estates.
And one finding that falls out of the same reading. Both estates’
deployment roots use the provider’s many-host SSH transport:
hosts = ...cluster_ssh_hosts, every node tried in order until one
connects (a deliberate 2026-07-27 fix). Combined with stampNode, which
records whichever node handled the request, deployment placement on these
estates is a function of which node answered SSH first. Two applies can
place two deployments on two different nodes with no declared intent and no
diff. Under this model that is directly visible: bound_to is derived from
where the bytes actually are, so the accident becomes a stated fact instead
of a hidden one. It is also an argument for the addressing model’s
observation that ssh_hosts ordering is an optimisation and must not be
built upon.
The swarm auto-pin, priced
Section titled “The swarm auto-pin, priced”Zero impact on both estates today (no swarm deployments). For any cluster
that does have one, the pin is added at the next render, which is an
update or move, not a background sweep. A deployment already carrying
its own deploy.placement.constraints is left alone:
ApplyDeployPlacement respects an app’s self-pin
(ce/modules/app/swarm_placement.go:58-63), and ApplyVolumePlacement
deliberately overrides it for provisioned RWO. Which of those two
precedents the binding pin follows is an owner ruling; see Q5.
What the model cannot do
Section titled “What the model cannot do”It cannot make a deployment unbound. There is no shared data path for
ordinary mounts (cells 2 and 7), so “move critical state onto a shared
provider” is not an executable verb. Until --storage is either made real
or retired, the only route from BOUND to UNBOUND is x-shc-volumes, which
no catalogue app uses. The model can therefore refuse and report, but the
owner’s escape hatch (“or becomes unbound”) does not exist yet.
It cannot see whether <StateHome> is itself on a share. An operator
who genuinely mounts /var/lib/shc from CephFS gets no credit: nothing
records or probes the deployment tree’s filesystem, and node.data_dir
exists in the schema and is never written. SHC over-binds, which is the
safe direction, but the one legitimate way to make compose mounts shared is
invisible.
It cannot locate an already-running unpinned swarm task. The auto-pin
prevents new ones; it does not retroactively find where an existing one’s
bytes are. For those, the honest answer is UNANCHORED and a manual
operator decision. Teaching SHC real swarm placement (docker service ps
carries the node) is adjacent work this model would benefit from but does
not require.
It does not fix driver_opts.device volumes pointing outside the
deployment dir. Backup discovers them; the render never sees them, since
they are a top-level declaration rather than a service mount source, so
DetectMountType’s SYSTEM allowlist never fires. They are host paths in
disguise and this model does not classify them. Called out because a
locality hole is worse when a locality model exists.
It does not defend against provider drift on an existing volume row.
EnsureVolume is get-or-create and does not update provider on an
existing row, so a config flip can leave the row’s provider and the
provider actually used at Bind divergent, and the stale value is what
the move classifier reads.
It does not give an operator a way to declare “I want this unbound and I accept the consequences”. Deliberately. Until a shared data path exists, such a declaration would be a request SHC cannot honour, and accepting unsatisfiable requests silently is the bug this document is largely about.
Slices
Section titled “Slices”Each slice is independently landable and independently provable. A slice is not done until its proofs run.
Slice 0: stop the bleeding, no model required
Section titled “Slice 0: stop the bleeding, no model required”Four fixes that are correctness today and that the model would otherwise inherit. None needs a schema change or a new concept.
- The move and dead-move classifiers stop trusting
deployment.storage. A provider that cannot be verified against a probed fact degrades toNotRebindablerather than being taken as gospel (ce/daemon/clone_move_adapters.go:117-120,ce/daemon/clone_dead_move_adapters.go:205-208). This alone closes the--storage nfssilent-loss chain by degrading it to backup and restore. shc cluster repairgains the deployment gate. It callsRemove(..., force=true)today with no gate at all (shc/modules/cluster/admin_router.go); at minimum it must apply the sameX400017NODDEPcheckshc cluster rmdoes.shc storage nfs|cephfs disableruns the in-use join thatDeleteProvideralready implements, instead of bypassing it (ce/modules/storage/shared_manager.go:245-249).- The storage documentation stops describing a system that does not
exist.
shc/docs/reference/cli/storage.md:8,16,233anddocs/architecture/modules/storage.md:161-167claim provider-held deployment state, a tenant/environment default resolution, and a swarm refusal namedX400044SWRLCL, none of which exists in either repo. Either fix the docs to describe HEAD, or land slice 4 and make them true; carrying a false safety claim is the worse of the three.
Proof: a unit test that an unverifiable nfs provider claim classifies
NotRebindable; a repair run against a fixture node carrying a deployment
refuses; a storage nfs disable with a deployment naming it refuses; a
docs-lint pass over the corrected pages.
Slice 1: one classifier, one durability property
Section titled “Slice 1: one classifier, one durability property”Collapse the six mount classifiers into core/volume, with backup, move,
dead-move, HUD and the binding predicate as consumers. Add the
durability: block to backup.yaml, three-source precedence,
critical by default, converter-covered mounts critical. Resolve the
absolute-host-bind disagreement between backup and move in backup’s favour
(critical, node-local, binds). Declare durability across the 73 catalogue
apps, starting with the 23 that already carry an exclude block.
Proof: a golden table over all 73 apps asserting every persistent
mount’s durability and its source, so a catalogue change that alters a
classification is a visible diff. Plus a fixture per structural rule
(tmpfs, anonymous, socket, read-only bind, pseudo-root, RUNTIME name
prefix) and a regression test that a converter-covered Postgres data
directory reads critical.
Slice 2: the predicate, derived, and visible
Section titled “Slice 2: the predicate, derived, and visible”Binding(D) → { state, bound_to, per-mount reasons } lifted out of the
daemon move adapter into a shared, target-optional form: targetNode == ""
means “tell me what this is bound to”. Then the two inspection verbs,
shc <stack> binding and shc node bindings <node>, rendering the same
reasons a refusal renders. Bound-ness appears as a column in shc ls.
Read-only throughout. Nothing refuses yet.
Proof: on a fixture cluster, shc node bindings names every deployment
and the exact mounts that bind it; a deployment whose only critical mount
is on a two-node shared provider reads UNBOUND; an unpinned swarm
deployment with a critical DATA mount reads UNANCHORED. Run against a
snapshot of both estates’ catalogue shapes, the eight bitspur deployments
all read BOUND to one node.
Slice 3: install makes UNANCHORED unrepresentable
Section titled “Slice 3: install makes UNANCHORED unrepresentable”The swarm auto-pin, reusing ApplyVolumePlacement’s mechanism and its
replicas: 1 clamp. The derived-versus-operator pin distinction. The four
refusals in the install table,
including the shc scale guard, which has no volume check today at all.
Proof: a two-node swarm cluster where shc install <db-app> --runtime swarm with no flags produces a service carrying
node.hostname==<installer> and replicas: 1; draining that node leaves
the task Pending: no suitable node rather than starting empty on the peer
and the peer’s volume is verified absent, not merely the task
observed; shc scale to 2 refuses; shc unpin refuses on the derived
pin and succeeds on an operator pin over shared storage. This is the proof
that needs a real multi-node swarm harness (tests/hetzner or
tests/proxmox), and it is the one proof in this document that cannot be
faked with fixtures.
Slice 4: node removal splits data loss from availability
Section titled “Slice 4: node removal splits data loss from availability”The four-way verdict table, applied to cluster rm, node forget,
cluster repair, and (via the addressing model’s slice 11) the provider’s
Delete. shc <stack> abandon --discard-local as the named escape.
Proof: removing a node holding a bound deployment refuses, names the
deployment and the binding mounts, and is not --force-able; the same
removal after shc move succeeds; removing the last node able to run an
unbound deployment warns and proceeds under --force; a cluster containing
one UNANCHORED deployment refuses every node removal with a message naming
the uncertainty rather than a deployment.
Slice 5: the dead-node path
Section titled “Slice 5: the dead-node path”stranded as a first-class deployment state, set when a bound deployment’s
node goes unreachable. Restart / redeploy / scale refuse on a stranded
deployment and say why. --presumed-dead’s classifier becomes the shared
one from slice 1. Per-critical-mount recoverability (last capture and
whether a restore path exists) printed before --discard-local proceeds.
Proof: a three-node fixture where killing the holder surfaces the
deployment as stranded naming the node; the node returning clears it with
no operator action; --discard-local prints, per mount, a capture age or
“NOT CAPTURED”, and the “NOT CAPTURED” case is produced by a real converter
with no matching restore job (the apps/redis/backup.yaml shape), not by
a synthetic one.
Slice 6: rule on --storage
Section titled “Slice 6: rule on --storage”The ruling is Q1, and Q1 is three rulings rather than one; see Q1 analysis. The REAL/RETIRED framing below is the original binary and is kept for the record; the proofs it names belong to rung 3, which the analysis recommends retiring. Under the analysis’s recommendation the REAL proof becomes rung 1’s (bytes verifiably under a probed store’s path) and the RETIRED proof becomes rung 3’s alone.
If REAL: --storage nfs|cephfs relocates MountsDir to
/mnt/shc/<provider>/deployments/..., which gives the dormant
storageapi.Provider interface a purpose, makes the storage documentation
true, makes cells 2 and 7 reachable, and makes “become unbound” an
executable verb for the first time. If RETIRED: the flag and the
deployment.storage column both go, and the classifiers lose a fallback
they should never have trusted.
Proof (REAL): a two-node cluster where a deployment installed
--storage nfs has its bytes verifiably under /mnt/shc/nfs, reads
UNBOUND, and shc move to the peer relocates with no data copy, with
the negative leg proven too: the same install onto a node whose client
mount failed refuses at install rather than recording a claim.
Proof (RETIRED): both estates plan clean with the attribute removed, and the classifiers’ provider input comes only from volume rows.
Slice 7: the filesystem probe
Section titled “Slice 7: the filesystem probe”Probe the deployment directory’s filesystem at render and persist the
result, so <StateHome> on a shared filesystem counts. The mechanism
exists: ce/modules/storage/doctor.go:88 isMountPoint already parses
/proc/mounts; it needs pointing at <deployDir>/mounts instead of at
provider rows. cephfs / nfs / fuse.ceph imply a shared anchor;
anything else node-local; a failed probe implies UNKNOWN, which binds.
Proof: a node with /var/lib/shc bind-mounted from an NFS export
reads its compose deployments as UNBOUND; the same node with a local
/var/lib/shc reads them BOUND; a probe failure reads UNANCHORED.
Slice 8: the vocabulary rename
Section titled “Slice 8: the vocabulary rename”hasBinding → hasVolume; “bound swarm DATA mount” → “provisioned swarm
DATA mount” throughout install_runner.go and the volume package. Purely
mechanical, no behaviour change, and it should land before slice 2
publishes bound as an operator-facing word.
Proof: the rename is complete: no occurrence of bound in ce or shc
means “has a block volume row”, asserted by a grep-shaped test in the
style already used for wire-shape ratchets.
Open questions: owner rulings
Section titled “Open questions: owner rulings”Product decisions, not engineering choices. Ordered by how much else depends on them.
Q1. --storage names three different questions. Which of the three is
real? This question was first written as a binary (is --storage REAL
or RETIRED?) and that binary is wrong. It collapses “WHICH NODE can serve
these bytes” into “WHICH BACKEND ON THIS NODE holds them”, which are
different questions with different answers. The owner’s correction, in one
sentence: “You still might have multiple storage providers that are
available for a specific node that you want to pick.” The three questions,
separated by what each does to anchor:
| rung | the question | effect on anchor | exists today? |
|---|---|---|---|
| 1 | which of THIS node’s disks holds this deployment (an NVMe beside a bulk HDD, a ZFS pool beside ext4) | stays { this node }; the deployment is still BOUND, you have merely chosen a disk | registry, CLI verb and provider implementation ship; the consumer was never written |
| 2 | which cloud block plane provisions this deployment’s volumes (hcloud, EBS, …) | { n : n.location == volume.location }, UNBOUND-IN-LOCATION; decision-table cells 3 and 8 | works: row-backed, fenced, RWO-clamped, declared via x-shc-volumes; the SELECTION surface does not |
| 3 | a shared filesystem (nfs / cephfs) holding a deployment’s data directory | every client node, truly UNBOUND; cells 2 and 7 | does not exist, and the schema cannot express it |
Each rung takes its own ruling: BUILD, FINISH, KEEP or RETIRE. It is entirely coherent for one to be finished, one kept and one deleted, and a single verdict on “the flag” cannot express three different answers. What is not coherent is the middle state all three share today: an unvalidated claim that changes nothing on rungs 1 and 3, has one undocumented negative effect on rung 2, and until slice 0 was trusted by two classifiers as though it were a fact. Almost every other answer’s shape depends on this one. Analysis, evidence and a recommendation: Q1 analysis: the three-rung reading.
Q2. Is the DATA-LOSS refusal truly non---force-able? “I don’t want
to ever be in a scenario where you can lose state” reads as yes. But an
operator with a physically dead disk needs some way out, and
--discard-local is the existing precedent for one that names what is
being abandoned rather than waving a generic override. The recommendation
is: non---force-able, with abandon --discard-local as the only door,
and slice 5’s recoverability report printed before it opens. What --force
means on the removal verbs TODAY (amnesia, not discard) and how that
narrows this question:
Node removal analysis.
Q3. Does durability belong in backup.yaml or in app.yaml?
backup.yaml is where the criticality reasoning already lives, in good
prose, written by app authors, in 23 of 73 apps; that is real continuity
and the recommendation. The cost is that a data-safety property lives in a
file named for a subsystem, which is a mild version of the overloading trap
this document otherwise argues against. app.yaml reads as a first-class
app property and is not backup-scoped, at the cost of splitting the
reasoning across two files.
Q4. Should install AUTO-PIN, or REFUSE? The recommendation is auto-pin,
because refusing makes stateful swarm impossible for all 73 catalogue apps
while pinning is a behaviour change that costs nobody anything and closes
the same hole. The counter-argument is that auto-pinning makes swarm
silently non-HA for stateful workloads and an operator may reasonably feel
SHC promised otherwise, which is why the recommendation is auto-pin
plus bound-ness visible in shc ls, in the same slice. Refusing is the
posture shc/docs/reference/cli/storage.md:233 already documents, so
ruling “refuse” also makes an existing doc claim true.
Q5. When the binding pin meets an app’s own deploy.placement, who
wins? Two precedents exist in one file: ApplyDeployPlacement respects
an app’s self-pin, ApplyVolumePlacement overrides it. The binding pin
protects data, which argues for override, but an app that pins itself
usually has a reason. Recommendation: override, and report it, but this is
a product call about whose intent is authoritative.
Q6. Is “unbound but nowhere left to run” a warning or a refusal? The
model says warning: it is an availability choice the operator is entitled
to make, and it is the only case where the gate softens. Confirming
matters because getting it wrong in the strict direction makes the last
node of a cluster unremovable, and in the loose direction makes a
--force that means two different things again.
Q7. Should there be a rebuildable durability class between critical
and ephemeral? Greenbone’s vt-data and huly’s elastic are both
“losing this costs hours, not data”. Today they classify critical and
therefore bind, which is safe and slightly over-strict. A third class could
warn-but-allow. Recommendation: two classes now; a third is a promise to
get the boundary right, and the boundary is exactly what nobody can agree
on.
Q8. Should shc cluster repair’s 24-hour auto-eviction survive at all?
It deletes deployment rows and node-local backup rows for any node quiet
for a day, with force hardcoded and no gate. That reads as designed for
phantom rows, not for live estates. Under this model it must at minimum
refuse for bound deployments; the open question is whether a
time-triggered destructive sweep belongs in the product at all.
Both premises in that framing are false and the question needs
re-asking: repair deletes no deployment row and no backup row, and its
24-hour leg is unreachable dead code because last_seen_at exists in no
migration and no ent schema. What it actually does, and the hazard that
replaces the stated one, are in
Node removal analysis.
Q1 analysis: the three-rung reading
Section titled “Q1 analysis: the three-rung reading”Analysis, not a decision. Q1 remains open and this document’s status line
still reads “awaiting owner ruling”. Written 2026-08-02 from four
independent evidence lanes and three judging lenses, re-verified
first-hand against ce origin/main at 182e4b8 (the clean worktree
/var/tmp/shc-s0/land-ce) and shc at 70fb6b45d. Every code claim below
carries a file:line. Neither production estate was contacted by this
analysis; live facts are attributed where they are second-hand.
The recommendation, per rung
Section titled “The recommendation, per rung”| rung | verdict | the one reason |
|---|---|---|
| 1: node-local store | FINISH, in three stages, with a probe as a hard precondition | The need is real and SHC has no mechanism for it (not one, not a bad one) and the two workarounds an operator would reach for either silently drop the mount from every backup or vanish on reboot |
| 2: block volume plane | FINISH | The mechanism is sound and armed on both estates; the SELECTOR is negative-only and evaluated after the branch that needed it, so the day an estate has two volume-capable connections a declared x-shc-volumes stack silently host-binds and comes up healthy on the root disk |
| 3: shared filesystem as a data path | RETIRE, scoped to the deployment-data-path claim | Its unique buy is HA-for-stateful, permanently ruled out; its non-unique buy does not even arrive, because MOVE-5 forces every compose node relocation onto backup+restore regardless of mount class and all 24 estate deployments are compose |
The panel split on rung 1 only, between BUILD and FINISH. Resolved as
FINISH, for two reasons. First, the strongest BUILD argument (that
SHC ships documentation promising shc install nextcloud --storage local-ssd) is stale: docs/reference/cli/storage.md was corrected at
76ffa6be4, hours before the panel read it, and now states plainly that
“registering a provider does not move any deployment onto it”. The promise
was withdrawn, which is the right call and which removes the deontic force
of the argument. What remains for rung 1 is the owner’s stated need on its
merits, which is enough, but is a thinner basis than BUILD implies. Second,
BUILD reads as one authorisation for one piece of work, and rung 1 is not
one piece of work: it is a cheap correctness fix that should land now, a
mandatory probe, and a path change that should wait for a real second disk.
Staging it is the honest verb.
The decomposition holds, with two amendments, and they lead
Section titled “The decomposition holds, with two amendments, and they lead”The separability critic’s objection outranks the per-rung verdicts, so it goes first. It found two things, and I confirmed both.
Amendment 1: the ladder is not a complete carve. There is a fourth
axis, it is live, and it is the one the binding model’s own escape hatch
depends on. backup.repo selects where a deployment’s ARCHIVE lives. It
is a config key, never a flag; it has its own anchor engine
(DeleteNodeLocalBackupRows decides node-locality by string prefix against
stateHome and deliberately KEEPS anything else, because “their data may
well have survived the node, and deletion is the irreversible direction”,
ce/modules/backup/ent_ops.go:214-223); and it defaults NODE-LOCAL:
repo: null at ce/core/config/default.yaml:188, whose own comment
instructs multi-node clusters to point it at “NFS: /mnt/shc-shared/backups
(requires NFS storage provider)”. A grep of both ops repositories’
hetzner/hel1/infra/main.tf for backup returns nothing, so on both
estates the archives sit on the same disk as the deployment they would
restore. That is second-hand (from the ops repos, not from a box) and
somebody should read it off a node before slice 5 is written, because if
it holds, this document’s entire consolation (“the cost of under-binding is
a restore”, --discard-local, slice 5’s recoverability report) is offering
a copy that dies with the original. This is a larger live exposure than
anything on the ladder, and it is not on the ladder.
Amendment 2: the rungs are separable for DELETION and for RULING, but
not for CONSTRUCTION, and that converts to a precondition rather than a
reversal. Separable for deletion, verified: the kind axis is a
three-valued enum with a TOTAL classification map and an AST guard
(providerTypeShared{local:false, nfs:true, cephfs:true},
ce/modules/storage/storageapi/types.go:38-68); only six non-test ce files
import the shared-FS packages; rung 2 lives in a different table
(volume / volume_attachment) reached by a different registry
(volumeapi builders) and a different grammar (x-shc-volumes), and
BuildVolumeBinder never queries store to choose a backend.
Separable for ruling: the three arms of ClassifyMountDecision are already
written and already disjoint, and the doctrinal exposure differs by the
entire trilemma: rung 1 spends no corner, rung 2 spends none because the
cloud substrate supplies the property, rung 3 spends both to buy back the
one that was given up. Three different answers cannot come out of one
verdict.
But not separable for construction, on four proofs:
CreateDeploymentFolderis the same function, byte-for-byte modulo error arguments, in all three providers:providers/local/provider.go:82-96,providers/nfs/provider.go:71-85,providers/cephfs/provider.go:60-74, eachdeploymentPath→MkdirAll(0o750)→for _, sub := range []string{"mounts","files","app"}. Finishing rung 1’s data path hands rung 3 its data path.- The registry cannot tell rung 1 from rung 3.
storecarries ONE cluster-widefield.Text("path")(ce/ent/schema/store.go:31);store_nodecarries only(store_id, node_id, created_at)with no per-node path override (ce/ent/schema/store_node.go:29-35); andshc storage link [name] [node-ids...]takescobra.MinimumNArgs(2)(ce/modules/storage/commands/link.go:53-55) whileService.LinkNodechecks only that the provider row exists and dedupes: no type check, no node-existence check, no probe (ce/modules/storage/service.go:497-514). Soshc storage add bulk --path /mnt/bulk -n n1 && shc storage link bulk n2 n3writes alocal-type provider linked to three nodes at one path: the database shape of a shared filesystem, expressible with rung 1’s verbs after rung 3’s packages are deleted. - Nothing probes the claim.
routes.handleAddbuildsCreateProviderInputand never sets capabilities (ce/modules/storage/router.go:179-186) whileCreateProvider’s own contract two hundred lines away says the caller must detect them (service.go:193-195).Service.ProviderInstance(the only function in either repo that constructs astorageapi.Provider, and therefore the only route tolocal.Detect/IsAccessible) has zero callers in ce (grep -rn ProviderInstance --include='*.go' .returns the definition and its doc comment, nothing else) and two in shc, both unit tests. So all sixteen interface methods are unreachable, not two. - The only validating storage-name surface reduces the whole namespace to
VolumeRestoreOverride{Provider, Provision bool}(ce/daemon/backup_volume_capture.go:531-585): provision a block device, or don’t. A named rung-1 store has no field it could be returned in.
The resolution is a constraint on execution, not a reversal of the
carve: a rung-1 FINISH must ship the probe that proves a named store is
genuinely node-local and singleton (at storage add AND at storage link,
on the target node) and must refuse a network filesystem type registered
as local. Without it, anchor = { this node } is an operator-typed
string with exactly the evidence quality slice 0 spent a night removing
trust from, and rung 1 silently re-admits rung 3 through the back door.
The pieces exist and none of them runs today: local.Detect,
Provider.IsAccessible, isMountPoint parsing /proc/mounts
(ce/modules/storage/doctor.go:87-107), and a transport that already POSTs
mount work to a remote node’s internal API.
The decisive factor
Section titled “The decisive factor”Rung 3’s unique buy (a stateful deployment surviving unplanned node loss
without a restore) is HA-for-stateful, the corner the trilemma
permanently gave up, and SHC would have to build the consistency domain
itself; rung 2’s equivalent property is handed over free by the cloud
substrate and rung 1 does not touch anchor at all. And rung 3’s
non-unique buy never arrives: relocateBlocked := nodeChanging && !swarm
sends every compose node relocation to StrategyBackupRestore regardless
of mount class (ce/modules/clone/move_strategy.go:294-345, MOVE-5), and
every deployment on both estates is compose, so a shared filesystem buys
nothing on move or drain on the runtime that is actually in production.
The vocabulary
Section titled “The vocabulary”One word meaning three things is what produced this question. Four axes now
exist, so four nouns are needed, and no word may read naturally for two of
them. Checked against this document’s own reserved set: bind
(addressing, unchanged and owned by the node-addressing model), binding →
provisioned (a mount carrying a volume row, slice 8), and
bound / unbound / bound_to / anchor (this model, deployment scope
only):
| axis | noun | operator surface | collision check |
|---|---|---|---|
| rung 1 | store | --store <name>, registered by shc storage add | --store is unused across ce’s command flag registrations. Carries no multi-node implication, which is the property that matters. A store is a singleton anchor by construction, once probed |
| rung 2 | volume | declared by the app in x-shc-volumes; backend selected by storage.volumes.provider (which needs a scope tier) and, if a per-deployment override is wanted, --volume-backend, never --storage | already the code’s word; slice 8’s hasBinding → hasVolume rename makes it unambiguous |
| rung 3 | share | reserved and unimplemented as a data path; a parse error in the --store slot, not a provider lookup. Legal only as a backup.repo target | new word, no collision |
| fourth axis | repository | backup.repo | already correctly named; the only change is that this document must stop assuming it is off-node |
--storage is retired as an operator word. Not because it is
worthless (it has exactly one live effect today, and any RETIRE ruling
must pay for removing it (see the corrections)) but because a single
argument position spellable by both local and nfs, told apart by a
string nobody probes, is the mechanism that produced this entire question.
The load-bearing rule is negative and should be written on the wall:
a shared kind must never be a legal value in the rung-1 slot.
The same defect already recurs one layer down and should be fixed in the
same breath: x-shc-volumes parses an unrecognised access token by
falling back to RWO, commented “the parser stays permissive”
(ce/core/volume/binding.go:130-147). So access: rwx (a request for
shared storage) silently yields a single-writer block volume, with no
warning and no lint. Two independent surfaces absorb a shared-storage
request into a node-local answer without a word. Fix the slot, not the
flag.
What this costs, including what the owner asked for and does not get
Section titled “What this costs, including what the owner asked for and does not get”Plainly, in his own terms: the escape hatch does not arrive. He said
“unless it’s moved or becomes unbound”. Under this recommendation
“moved” works and “becomes unbound” does not, not for an ordinary
compose deployment’s data directory, which is every deployment on both
estates and every stateful app in the catalogue. Retiring rung 3 declines
to build the one shape that would have made cells 2 and 7 reachable, so a
BOUND deployment’s route out stays: shc move, which copies.
What he gets instead, and it should be stated as an obligation rather than a consolation:
- Rung 2 becomes the real escape hatch, and it already works. A
provisioned block volume detaches and reattaches with no copy, fenced by
volume_attachment.lease_generationand RWO-clamped at install. Both estates already carry exactly one volume-capablehetznerconnection, so the substrate is present and configuration-free. The gap is the catalogue:x-shc-volumesappears in zero of 73 apps. That is one compose extension block per stateful app, not a new consistency domain, and it is the follow-on that makes this retirement honest. - The ceiling gets written down instead of engineered around.
moveanddrainstage through a LOCAL archive (MakeTempDir("shc_clone_")→<StateHome>/staging) before shipping it, and phase order is backup → stop → recover → uninstall, so source and target must each hold dataset plus archive simultaneously. On a single 610 GB disk that is a hard ceiling somewhere near 45% of the disk, and above it a deployment cannot be relocated at all. That is a real capability limit with a number attached and it belongs in the operator docs in plain words. --discard-localstops being a coin flip, but only if slice 5’s recoverability report also reports whether the archive is on the same disk as the thing it would restore. See the fourth axis.
The rest of the bill:
- Rung 1 stage 3 is pervasive but bounded. Twelve non-test inline
re-derivations of
filepath.Join(<x>Dir, "mounts")(daemon/clone_move_adapters.go:648,modules/app/install_runner.go:3023,modules/app/post_install_hooks.go:359,modules/adapter/commands/migrate_from.go:276,modules/backup/restore.go:384and:471,modules/backup/service.go:1001,modules/backup/cli_runner.go:161,:388,:549,:1866,:2084), plus the two resolver families (core/paths.Paths.MountsRootandapp.DeploymentDir), plus six name-based invariants that encode “mounts is a child of the deployment dir” by string:install_runner.go:2081(the keep-data survivor set),backup/archive.go:309(separatelyArchivedDirs),doctor/diagnoses/database_orphaned_dirs.go:38, and threeinfo.Name() == "mounts"walk tests atrestore.go:1033,:1229,:1292. Plus shc’s hardcoded acme glob. - One ordering constraint that is not negotiable.
PersistentMountsFromComposeclassifies SHC data by string prefix (mountsPrefix := filepath.Join(deployDir, "mounts") + os.PathSeparator(ce/daemon/clone_move_adapters.go:648)) and anything absolute outside it falls intohostBinds, which the function’s own comment declares “classified ephemeral; SHC data never lives there by convention”. That site must be converted before any install can write outside the default root. On compose the failure is a placement lie rather than byte loss (MOVE-5 copies anyway, andcaptureHostBindsdoes tar absolute binds); on swarm it isStrategyEphemeralOnlyand the bytes stay behind. - Two costs of RETIRE that must not be skipped. (i)
refuseIfInUsejoins BOTHentvolume.ProviderInandentdeployment.StorageIn(ce/modules/storage/service.go:423-469) to gateshc storage rm,DisableNFSandDisableCephFS. Because the entVolumeCHECK forbids a shared-FS volume row, the deployment leg is the only live leg for nfs/cephfs. A rung-1 FINISH keeps that leg alive under a new flag name; if both retired, the guard would silently lose its subject. (ii) Slice 0 shipped two AST guards whose subject set is derived fromproviderTypeShared. DeletingProviderTypeNFS/ProviderTypeCephFSmakes them vacuous rather than red, the fifth “guard is green while the thing it guards is broken” of this session. They must be replaced with an assertion that the shared set is EMPTY and that nolocalprovider is linked to more than one node. - Docs are doubled.
docs/andwebsite/src/content/docs/are independent copies with no sync target, so every page is edited twice.
The build / deletion list
Section titled “The build / deletion list”R1a: validate, and finish the docs (land now; no path change, no
migration). Install refuses a --store name with no store
row, or a row not linked to the target node, reusing the join
providerUsage already implements (ce/modules/storage/service.go:446-469).
Today install stores the string verbatim (ce/modules/app/service.go:475-477;
grep -ci storage ce/modules/app/install_refusal.go = 0) while
shc move / shc recover refuse unknown names with X400865VOLSOV, two
ends of one flag disagreeing about its domain, with only the
destructive-adjacent end right. Docs: docs/reference/cli/storage.md is
already correct as of 76ffa6be4, but
docs/guide/operations/storage-providers.md:273-295 is still entirely
fictional: its five-input algorithm names an app-declared
requires.storage whitelist, a whitelist-checked per-stack --storage
override, node affinity, a capability filter and a cluster default, and
grep -rn 'requires.storage\|RequiresStorage' --include='*.go' over both
repos returns zero. requires.storage also appears in
docs/app-development/creating-apps.md,
docs/guide/troubleshooting/README.md, and (as a live shell demo on the
public site) website/landing/arch/where-it-lives/index.html:534-537
($ shc install nextcloud --storage nfs → “the app’s requires.storage
whitelist still has to allow it”) plus its landing-archive twin.
R1b: the probe (precondition for R1c, valuable alone). Run
local.Detect + IsAccessible on the TARGET node at storage add and at
storage link, over the existing remote transport. Persist the probed
filesystem type. Refuse a network filesystem registered as local; refuse
(or add a nullable per-node path column to store_node, the
cheap append-only ADD COLUMN shape) a local provider linked to more than
one node. Gives Service.ProviderInstance its first production caller and
the sixteen dormant interface methods their first reachable path.
R1c: the resolver (defer until an estate has a second disk). One
MountsRootFor(tenant, stack, env) reading the manifest’s store name →
the provider row’s path. Convert clone_move_adapters.go:648 FIRST, then
the eleven other inline joins, then the six string invariants. Design B
(relocate mounts/ only, leaving compose.yaml / manifest.json / app/
under the state home) keeps GetStackFromDisk and the manifest findable
and avoids the ~140-site whole-DeploymentDir variant. Follow-on: the
scheduler’s per-node disk is statfs("/")
(ce/subsystems/impl/scheduling_source.go:76, :104), so a relocated
deployment would still be billed against the root filesystem.
R2a: make the block selector positive (~15 lines, one function). In
ce/daemon/volume_binder.go:357-407, hoist the storageDefault check
ABOVE the auto-select block. Today the default: arm of that switch warns
X400327VOLAMB and returns nil at :398-400, before the flag is read
at :404-407, so on a two-candidate estate the flag can never break the
tie it sits next to, and the error’s own text tells the operator to edit
storage.volumes.provider instead. If the name matches a live
volume-capable connection or a registered block impl → select it; if it
names a store or a share → decline, preserving today’s kill-switch exactly;
if unrecognised → refuse LOUDLY rather than silently host-binding. Every
recogniser already exists.
R2b: thread the resolved provider into the provision (two files).
DaemonStorageOverrideResolver genuinely validates a provider name and
refuses unknowns, then resolveProvisionedOwners passes it to
ProvisionForRestore, which builds its binder as a.binderFn(ctx),
wired at ce/daemon/services_backup_config.go:140-142 as
d.BuildVolumeBinder(ctx, paths.Default.StateHome(), ""), and uses the
provider argument only inside error strings
(ce/daemon/backup_volume_capture.go:292-299). So shc move --storage aws-ebs-volume validates, succeeds, and re-provisions on the global
backend. A validation chain terminating in a call that ignores its argument
is worse than no validation.
R2c: two safety rails. (i) Service.Scale
(ce/modules/app/service.go:859+) has no volume awareness at all, so a
routine shc scale defeats the RWO pin plus replicas: 1 clamp the
install stamped at install_runner.go:3153. (ii) A bound SWARM stack
cannot be updated: install_async.go:522-533 deliberately resolves no
binder (Bind is not idempotent), so shc update returns X400324VOLSWMNB
whose registered template reads “no volume backend is configured”, which
is false, and sends the operator to check a token that is fine.
R3: retire the shared filesystem as a DEPLOYMENT DATA PATH, scoped.
Goes: the case "nfs","cephfs" arm of DaemonStorageOverrideResolver
(ce/daemon/backup_volume_capture.go:529-542); the isSharedFSProvider
arms of both move classifiers and the X503085SHFSMT warning; --store nfs|cephfs becomes a parse error at install and at move/recover; the
public prose above. Does NOT automatically go: the ~6,250 non-test Go
lines of node-side mount machinery (modules/storage/ceph 2230,
providers/nfs 990, providers/cephfs 850, modules/storage/nfs 508,
shared_manager.go 505, shared_manager_cephfs.go 331,
commands/managed.go 499, swarm_label.go 103,
daemon/cephfs_provisioner.go 102, daemon/services_ceph_peer.go 138),
ten CLI verbs and ~10 pkg/apiclient methods. Whether those survive is a
separate call keyed on backup.repo, because a shared ARCHIVE target
is a share used the way a share is safe (one writer at a time, restic’s
own locking, no live-writer semantics) and default.yaml:182-183 already
documents it. Deleting the substrate to satisfy a doctrine about live data
paths would break the documented archive path the doctrine never spoke to.
Free, and worth doing whichever way this rules.
diagnoseStores does os.Stat(rec.Path) at
ce/modules/storage/doctor.go:53 BEFORE isMountPoint at :67, and
isMountPoint is the safe one, since it parses /proc/mounts. On a hung
hard NFS mount shc doctor blocks inside the one check meant to notice the
outage. One-line reorder.
The migration
Section titled “The migration”None is required, in either direction. deployment.storage already
exists, is already written, and is the field rung 1 needs; reusing it under
the name --store costs zero migration. The ent field name stays
storage; the concept is a store; one comment reconciles them.
Under RETIRE, the column must be RETIRED IN PLACE and never dropped.
The house precedent is three lines below it in the same struct:
launcher at ce/ent/schema/deployment.go:73-78: “RETIRED IN PLACE
(pre-1.0) … No code reads or writes the column; it stays in the schema only
because migrations are frozen (atlas.sum). Always NULL on new rows.” A
naive drop is not merely expensive: SQLite has no DROP COLUMN for atlas,
so it emits a table rebuild, and applyOne runs a whole migration file
inside ONE BeginTx, where SQLite documents PRAGMA foreign_keys as a
no-op. An evidence lane proved the consequence empirically against modernc
sqlite: service / host_mapping / port_mapping / integration rows
cascade-deleted, backup.deployment_id NULLed. Six tables hold FKs to
deployment(id). Do not go near it.
The only additive change contemplated is R1b’s nullable per-node path on
store_node, the one-line ADD COLUMN shape
(migrations/20260730032524_node_cpu_cores.sql is the precedent), never a
rebuild, never a flatten.
shc’s hand-written DDL mirrors need nothing. Both are partial subsets
(shc/tests/unit/subsystems/impl/stack_reconcile_ports_test.go:39,
shc/tests/unit/modules/doctor/diagnoses/database_test.go:27) and neither
ever carried storage.
NO BACKCOMPAT is satisfied trivially. Three lanes independently read
both estates read-only and agree: shc storage ls returns [] on both
(the store table is empty; not even a local row),
shc volume ls returns none, all 24 deployments carry storage IS NULL in
both the DB and the on-disk sidecar, and all 24 are runtime: compose.
Zero storage = attributes across all three ops repos; zero
x-shc-volumes across all 73 catalogue apps; no shc_storage_provider
terraform resource exists at all. Whichever way Q1 lands, it breaks zero
live deployments. One forced exception survives regardless:
shc/terraform/provider/internal/provider/state_upgrade.go holds the
frozen decode-only v0 schema, and deploymentSchemaV0() must keep
storage or every apply against pre-v1 state fails to decode. That is
Terraform’s state-upgrade contract, not SHC’s.
What this unblocks, and how each downstream question changes shape
Section titled “What this unblocks, and how each downstream question changes shape”- Slices 1, 2, 3, 4: unblocked unchanged. The predicate, the
classifier collapse, the auto-pin and the removal split were never
waiting on
--storage; they were waiting on Q1 only because Q1 decided whether an UNBOUND destination existed. - Slice 5 (the dead-node path): reshaped, and this is the sharpest
change. The recoverability report gains a mandatory second leg:
per-critical-mount capture age AND whether the archive is node-local.
With rung 3 retired and
backup.repounset,--discard-localon a live estate today is not “four hours versus everything”; it is “everything”, and the prompt cannot say so. - Slice 6: becomes three sub-rulings instead of one. Its REAL/RETIRED
proofs both need rewriting: the REAL proof was written for rung 3
(bytes verifiably under
/mnt/shc/nfs, move with no data copy) and becomes rung 1’s (bytes verifiably under the store’s path, a same-node storage re-home, an install onto an unprobed store refused). - Slice 7 (the filesystem probe): repointed and promoted. It was
written as a late, optional probe of
<deployDir>/mountsat render, to give credit to an operator who mounts/var/lib/shcfrom CephFS. Under this recommendation it becomes R1b: a probe atstorage add/storage link, on the target node, and a precondition of rung 1 rather than a nicety. Its shared-filesystem-implies-shared-anchor arm inverts: acephfsfstype under a registered store becomes a REFUSAL, not credit. - Slice 8 (the vocabulary rename): grows by one word and should still
land first.
hasBinding→hasVolumeas written, plus--storage→--storeand the reservedshare. - Q2 (non-
--force-able data-loss refusal): unchanged in principle, conditional in practice. The recommendation stands, butabandon --discard-localis only an honest door if the archive is off-node.backup.repobecomes a precondition of Q2’s answer, not a detail. - Q4 (auto-pin versus refuse): strengthened toward AUTO-PIN. The
counter-argument was partly that “refuse” would make an existing doc
claim true (
storage.md:233, “Swarm-mode stacks require a shared provider”). That sentence has since been deleted as false, and rung 3’s retirement removes the destination a refusal would have sent the operator to. Refusing now means refusing stateful swarm with nowhere to go. - Q3, Q5, Q6, Q7: unchanged. None depends on which storage axis is real.
- Q8 (
cluster repair’s 24-hour sweep): worse than it read. It deletes node-local backup rows, and withbackup.repounset that is nearly all of them on both estates. - Not unblocked, and it should be a condition of ruling rung 3: the
trilemma is cited by two architecture docs
(
docs/architecture/cluster/swarm-contention-map.md:493,swarm-quorum-and-health.md:605) as living indocs/guide/getting-started/concepts.md, and a grep of that file fortrilemma|pick two|high availab|stateful|node loss|reschedulreturns zero hits over all 328 lines. A ruling that is only ever cited and never stated will be re-litigated. Write the paragraph in the same commit.
The weakest point in this recommendation
Section titled “The weakest point in this recommendation”Rung 1’s stage 3 rests on a demand claim nobody can establish. No node
on either estate has a second disk (three lanes’ live lsblk: one 610.4 GB
sda, sda1 ext4 at /, no spare device on any of six nodes). No ops
repo sets a storage attribute. No catalogue app needs one. store
is empty on both estates, which means the rung-1 registry may never have
executed outside unit tests, and nobody grepped the bats/e2e suites to
check. The affirmative case is the owner’s stated need plus a promise the
docs have since withdrawn, not an observed operator hitting a wall.
What would overturn it: evidence that shc storage add has never been
exercised anywhere outside unit tests, combined with the owner confirming
he was describing a shape he wants rather than one he has. That collapses
the ruling to R1a + R1b (validate, probe, document) with R1c indefinitely
deferred, which is RETIRE-in-effect for the relocation half. Note the
asymmetry: if the single-disk finding is WRONG, the case gets stronger, not
weaker. R1a and R1b are worth landing under either outcome, which is the
argument for staging.
A second, structural weakness, stated because two judges raised it independently: the rung-3 RETIRE leans on a doctrine that is not written where it is cited, and the distinction that spares rung 2 while killing rung 3 (the trilemma forgoes BUILDING a consistency domain, it does not forbid one the substrate hands you free) is this analysis’s reconstruction of intent, not a quoted ruling. If the owner’s actual line is narrower (“don’t build a distributed filesystem ourselves”), rung 3 still loses on MOVE-5 alone but loses as a COST argument rather than a doctrinal one. If it is broader (“no stateful failover, period”), rung 2’s location rebind is already over the line and its FINISH is wrong.
Corrections to the record
Section titled “Corrections to the record”These came out of the evidence lanes and contradict facts asserted elsewhere tonight. They are load-bearing beyond Q1.
--storageis NOT inert on rung 2. The claim that “every production call site ofBuildVolumeBinderhardcodes an EMPTY storage” is wrong. Four DIRECT calls do (services_seams_stack.go:729,services_backup_config.go:141,volume_ops.go:430), but the mainline install goes through the seam:ResolveVolumeBinder(ctx, stateHome, req.Storage)atce/modules/app/install_async.go:139, whose provider isd.BuildVolumeBinderregistered atce/daemon/services_extra.go:235-236. So--storageis today a live negative kill switch:if sd != "" && sd != providerName { return nil }(ce/daemon/volume_binder.go:404-407) declines the block binder and forces the host-bind path. It is the flag’s one real behaviour, it is undocumented as such, and retiring the flag silently re-enables block provisioning for anyone using it that way.- There is no
shc restore --storageflag. Exactly three registrations exist in either repo:shc install(ce/modules/app/commands/install.go:152),shc recover(ce/modules/app/commands/stack_lifecycle.go:641), andshc clone/shc move(ce/modules/clone/commands/main.go:300viabindCloneFlags).DaemonStorageOverrideResolver’s false-premise problem is real, but its surface is move/recover, which matters because retiring the flag at install does not by itself retire that path. - The dormancy is much wider than “
BasePath()andCreateDeploymentFolder()have zero production callers”.Service.ProviderInstanceis the only constructor of astorageapi.Providerin either repo and has zero callers in ce. Nothing in production ever holds a Provider, so all sixteen interface methods are unreachable, not two.storage.MountPathis a second, independent provider→path mapping with zero callers anywhere. docs/reference/cli/storage.mdwas corrected at shc76ffa6be4and now states that registering a provider moves nothing. Any argument resting on that page still promising--storage local-ssdis reading a superseded revision. Butdocs/guide/operations/storage-providers.md:273-295and the public landing page still describe the fictional five-input selector, andrequires.storage(its first input) appears in zero Go files in either repo. That is the same shipped-promise-with-no-implementation shape that taught two classifiers to trust a label.shc storage nfs|cephfs disableno longer bypasses the in-use guard. ce182e4b8addedrefuseIfInUseand placed it BEFORE the node teardown. An assertion to the contrary is describing a state that was true earlier the same night.- Slice 0 is on ce
origin/main, not in the working checkout./home/admin/selfhosted-cloud/ceis 128 commits behind at2815168and still shows the pre-slice-0 classifier. Anyone re-verifying there will read the oldcase "nfs","cephfs": → Rebindablearm and reach a different conclusion. Use a fresh worktree offorigin/main. node.default_storeis not a dead COLUMN; it is a phantom. It appears in no ent schema, noent/migrate/schema.goand no migration SQL, surviving only as a Go/wire orphan rendered byshc node info.docs/architecture/DRIFT-REGISTER.md:625describes it as a column. It is also, notably, the exact shape a node-scoped store default would need, an axis this analysis priced nowhere, because the config scope ladder has Global/Tenant/Environment/Stack/Deployment and no NODE tier.- Estate counts. Both estates run twelve deployments, not eight:
the eight declared in
corp.tfplus fourmaster-tenant system apps (traefik, keycloak, postgres, openobserve) that no ops repo declares. Slice 2’s proof text (“the eight bitspur deployments”) will under-count.x-shc-scaleappears in 48 distinct apps (63 occurrences), not “63 of them”; thex-shc-volumes= 0 of 73 figure is confirmed.
Node removal analysis: does force-remove back up, then delete?
Section titled “Node removal analysis: does force-remove back up, then delete?”Analysis, not a decision. This document’s status line still reads
“awaiting owner ruling”. Written 2026-08-02 from three evidence lanes and
three judging lenses, re-verified first-hand against shc 17357d5fa and ce
at the commit shc actually pins: 182e4b832582 (go.mod:8), not the local
ce checkout, which is an ancestor. Every claim below carries a file:line.
Neither production estate was contacted by this analysis; the handful of
live facts are attributed as second-hand where they appear.
The owner’s question, put thinking aloud and explicitly uncertain:
“I would say if you force remove a node, it deletes the deployments too, but it backs them up first. Or not. I don’t know. I guess so. I don’t know.”
The recommendation
Section titled “The recommendation”shc node forget gains the occupancy gate shc cluster rm already
ships, and neither verb ever deletes a deployment row or takes a backup.
--force keeps the meaning it already has everywhere else in SHC:
“remove the node anyway and leave the work stranded”, and never acquires
a second, destructive one.
What the operator types, end to end. Retiring a node that still answers:
shc node drain <node>, then shc node forget <node> --yes. Removing a
node that is gone: shc node forget <node> --yes, which refuses naming the
stack count, then shc node forget <node> --yes --force, which removes the
node and leaves its deployment rows stranded, after which
shc move <stack> <stack> --node <peer> --presumed-dead recovers each one.
The one reason: force-remove does not delete deployments today, so the
owner is not choosing between “delete” and “delete plus backup”; he is
choosing whether to ADD deletion. rqliteNodeRemover.Remove
(subsystems/rqlite_adapters.go:179-219, read end to end) evicts the raft
member, writes UpdateState(nodeID, "removed"), may demote the engine to
plain sqlite, and sets exactly two fields on its result: NodeID and
Warnings. It touches no deployment row, stops no container, runs no
docker node rm. Once the question is reframed that way, every consequence
that makes deletion look necessary (a dangling DNS record, a stale Traefik
backend, leaked quota, a stack frozen at running) turns out to be fixable
by three predicates and a display change, none of which delete a row.
How the panel split, and how it resolved. All three lenses converged on
four points: retain the rows, delete the false counters, fix the
presumed-dead foreclosure first, and land the DNS predicate in the same
change. They split on whether the verb should REFUSE when the node is
reachable: the 3am-operator lens said no (a verb whose behaviour you must
reason about at 3am is a bad verb), the other two said yes. Resolved by
separating the two things “branch” was being asked to do: the destructive
behaviour does not branch at all (nothing is ever destroyed, on any path),
and the refusal branches on OCCUPANCY rather than on REACHABILITY.
Occupancy is a raft-replicated DB fact that is equally available whether the
node is alive, dead, or partitioned; reachability is a 2-second TCP dial that
shares a failure domain with the partition it exists to detect. Branching on
occupancy gives the honesty lens its refusal and the 3am lens its single
predictable rule, and it costs no new classifier because
X400017NODDEP already ships (shc/modules/cluster/admin_router.go:171-183).
This is also the shape the owner has already ruled for.
docs/guide/operations/node-rotation.md:170-177 records it verbatim: “a
destroy must refuse and name what is in the way; it must never auto-relocate.
Relocation stays shc node drain”, together with the already-scoped
mechanism on both sides: shc node forget gaining X400017NODDEP “with an
explicit --force for the already-gone machine”, and the provider gaining
shc_node.on_destroy ("refuse" by default, "force" per node). That
second half matters, because “a refusal breaks tofu destroy” is the
strongest objection to gating forget, and the answer to it is already
designed. Backup-then-delete is auto-DESTRUCTION on a destroy path, which is
further from that ruling than the auto-relocation it already forbids.
Does it branch on reachability? No, and that is deliberate
Section titled “Does it branch on reachability? No, and that is deliberate”Nothing about what is destroyed depends on whether the node answers, because nothing is destroyed. What the verb SAYS depends on occupancy, which is knowable without a probe. Case by case:
| situation | what happens |
|---|---|
| Node dead, occupied | forget refuses with the stack count; --force removes the node and strands the rows. Recovery per stack is shc move … --presumed-dead, which requires the fix in the next section. |
| Node alive, occupied | Identical refusal, and here the named remedy is real: shc node drain <node> genuinely relocates, then forget succeeds with no override at all. --force still works for the operator who means it, and strands rather than deletes. |
| Node partitioned but alive | Identical, because the gate never asks. This is the case that decides the design: under a reachability branch, a partition fails both the durable unreachable stamp (roughly six seconds, a 2-second tick at modules/health/crons.go:75 against a default threshold of 3) and the fresh probe together, so the verb takes the dead branch against a node serving production traffic. Under an occupancy branch there is no misclassification to make. |
| Automatic 24-hour sweep | There is none; this is a correction, see below. shc cluster repair’s stale leg reads m.LastSeenAt, and last_seen_at exists in no migration and no ent schema. Only the unreachable leg can fire, and only when an operator runs the command. |
tofu destroy of one node | The provider’s Delete runs shc node forget <name> --yes on the BOOTSTRAP over SSH (shc/terraform/provider/internal/provider/node_resource.go:1063) against a machine that is alive by construction, since the VM is destroyed strictly after (node_resource.go:1023-1026). It hits the refusal and must send on_destroy per the standing ruling. Worth flagging separately: runtime, nic and name carry RequiresReplace (node_resource.go:256-271), so editing runtime on a healthy production worker force-forgets a machine that is not being destroyed at all and then re-joins the same box. |
tofu destroy of the whole estate | Every node is occupied and every removal needs the override, which is correct: a whole-estate teardown is exactly the case where the operator does mean it. This is also the one place where retaining rows is actively harmful, because the retained row is what keeps a dangling A record unreapable, which is why the DNS predicate below is not optional. |
The owner’s instinct, in his terms
Section titled “The owner’s instinct, in his terms”The “delete” half does not survive, and the reason is simpler than the
code: SHC would be deleting the only description of the deployment it has
left. Deleting a deployment row cascades away service, host_mapping,
port_mapping, integration in both directions, and child deployments:
six inbound foreign keys, all ON DELETE CASCADE
(migrations/20260727022830_auto.sql:291,367,386,387,561,627). Exactly one
inbound key is ON DELETE SET NULL: backup (:85). So the schema was
deliberately built so the backup row outlives its deployment, but per lane
2’s live read, zero of the twenty-four production stacks across both estates
has a backup, and bitspur’s shc backup ls returns literally “no backups”.
host_mapping is the database’s only record of what hostnames a deployment
published; its only other copy is the manifest sidecar on the disk being
removed. Deleting the rows is the estate losing its own inventory at the
exact moment it needs it to rebuild.
The “backs them up first” half does not survive either, and this is the
part worth the owner’s attention, because it is not really about node
removal at all. Two of the three reasons are addressed in the next section
as a separate recommendation. The third is specific to this verb: a backup is
only possible when the node answers, and the product has already ruled on the
case where it does not. shc move --presumed-dead’s own header states it:
“NO BACKUP LEG: the source is dead, so no pre-move backup can be taken”
(modules/clone/move_presumed_dead.go:32-34). So “always back up first”
cannot be an invariant; it degrades to “attempt, and refuse when you cannot”,
and the population where it CAN succeed is precisely the population where SHC
already has a better verb. For a live node the owner gets the backup-first
relocation he is describing, done properly (backup, ship, bring up on a
peer, then uninstall the source) and it is spelled shc node drain.
Put plainly: the instinct is right that a node should not be removed while its work is silently abandoned. It is wrong only in reaching for deletion and backup as the mechanism, when the mechanism is a refusal that names a verb that already exists.
Prerequisite: removal currently destroys the dead-node recovery verb
Section titled “Prerequisite: removal currently destroys the dead-node recovery verb”This is a live bug, it is independent of any ruling, and it must land before the dead-node half of the recommendation works at all.
Remove writes cluster_state = "removed"
(shc/subsystems/rqlite_adapters.go:199). PresumedDeadVerdict refuses
unless the state is EXACTLY unreachable:
if state != string(nodemod.ClusterStateUnreachable) at
daemon/clone_dead_move_adapters.go:114-117. The lookup that feeds it,
GetByName, carries no tombstone filter (modules/node/repository.go),
unlike List, which does. So the tombstoned node resolves and the refusal
fires:
presumed-dead refused, node {node} cluster_state is {state} not unreachable, if the node is up use a plain move
modules/clone/errors.go:240-243
Net: shc node forget <dead node> permanently forecloses
shc move <stack> <stack> --node <peer> --presumed-dead, the one path that
rebinds surviving RWO volumes off a dead box, and it does so by telling the
operator the node might be up, about a node the cluster has just tombstoned.
The --presumed-dead flag’s own help text states the precondition that the
removal destroys: “refused unless the node is registry-unreachable AND a
fresh probe fails” (modules/clone/commands/main.go:270-271).
The fix is two lines: accept removed as satisfying GATE 1a. A node the
cluster has tombstoned is strictly more dead than one merely stamped
unreachable, and GATE 1b’s fresh probe (DeadNodeTCPProbe, 2-second budget,
clone_dead_move_adapters.go:145-158) still guards the live case. Route
touched: all three, since all three call Remove.
★ backup.repo defaults to the disk it would restore: a separate recommendation
Section titled “★ backup.repo defaults to the disk it would restore: a separate recommendation”This is bigger than the node-removal question and should be ruled on independently of it. It is stated here as its own recommendation because it is the reason the owner’s backup instinct cannot be implemented, but it would remain true if node removal were never touched.
repo: null at core/config/default.yaml:188. The config layer does not
resolve that default; the value is materialised at boot by the backup_mount
subsystem, and resolveDefaultBackupRepo() reads the ENVIRONMENT, not the
paths package:
stateHome := os.Getenv("SHC_STATE_HOME")if stateHome == "" { return "", ""}return filepath.Join(stateHome, "backups"), defaultBackupPassword()subsystems/impl/backup_mount.go:416-425. And immediately below it:
func defaultBackupPassword() string { if pw := os.Getenv("SHC_BACKUP_PASSWORD"); pw != "" { return pw } return "shc-default-backup-password"}:431-437, in the AGPL public repo.
Per lanes 1 and 2’s live read (second-hand; not re-verified here), neither
ops repo overrides backup.repo, SHC_STATE_HOME=/var/lib/shc comes from
shc/shc.service:36, /var/lib/shc/backups is a real restic repo on both
estates, and /var/lib/shc is the same filesystem as / on all six nodes.
Two consequences, both independent of this ruling:
- The archives sit on the disk they exist to restore. On a single node
that is a defensible default and the config comment says so. On a
three-node estate it means the backup of every stack on
compose01is oncompose01. - They are encrypted with a password published in public source. Any archive in a default repo is readable by anyone who can read ce.
The recommendation: make an off-node backup.repo a first-class part of
estate bring-up rather than an optional tuning step, and treat the hardcoded
password as a defect in its own right. The remedy verb exists:
shc config set backup.repo <value> (modules/config/commands/set.go:22).
Until then, any feature that captures a backup as a safety net before a
destructive operation is capturing it onto the thing being destroyed, which
is the “success not earned” class: the operator is told there is a backup, and
acts on it.
The exact strings and counters that must change
Section titled “The exact strings and counters that must change”Every verb named below was checked against the live cobra tree, not from
memory. shc cluster registers init, status, nodes, reconcile, logs, ping,
cert, fabric (ce modules/cluster/commands/main.go:285-306) plus join,
leave, repair, rm, mesh (shc modules/cluster/commands/commands_edition.go:34-40).
There is no shc cluster forget and no shc cluster drain. shc node
registers status, token, join, leave, forget, drain, undrain, swarm-tokens
(modules/node/commands/fleet.go:27-34). shc move
(ce modules/clone/commands/main.go:198), shc recover
(ce modules/app/commands/stack_lifecycle.go:491) and shc config set
(ce modules/config/commands/set.go:22) all exist.
Four never-assigned result fields, not one. git grep -nE "(DeploymentsEvicted|BackupsEvicted|SelfRemoval|MeshAddressReclaimed)" over shc
returns only copy-outs and struct tags; the same grep at ce 182e4b832582
returns only declarations. Remove sets NodeID and Warnings and nothing
else, so all four are structurally always zero.
| what | where | action |
|---|---|---|
SelfRemoval, MeshAddressReclaimed, DeploymentsEvicted, BackupsEvicted declared | ce modules/cluster/service.go:2683-2689; ce modules/cluster/admin_router.go:77-80 | delete |
copied out on cluster rm | shc modules/cluster/admin_router.go:221-224 | delete |
copied out on node forget | shc modules/cluster/lifecycle_router.go:364-365 | delete |
rendered "%d deployments evicted" / "mesh IP … reclaimed" / "self" | shc modules/cluster/commands/admin_fleet.go:279-287 | delete |
rendered "forgot node %q (mesh ip %s reclaimed)" | shc modules/node/commands/forget.go:86-89 | delete |
| wire types | shc pkg/apiclient/types_gen.go:995-996, 2270-2273; CLI result structs modules/cluster/commands/admin_fleet.go:48-50, modules/cluster/lifecycle_router.go:106-107, modules/node/commands/forget.go:19 | regenerate / delete |
A correction on how this is described. Two of the three cases put to the
panel framed this as an operator being told a false eviction count. That is
not what ships: every render is guarded (if r.DeploymentsEvicted > 0
(admin_fleet.go:285), if r.MeshAddressReclaimed != "" (:282), if r.SelfRemoval (:279)), and the JSON fields are omitempty. No false
count is ever printed. The shipped defect is one of omission, dead weight,
and doc comments, which is why the fix is deletion rather than assignment. It
still matters: a > 0 guard makes “no eviction happened” indistinguishable
from “zero deployments”, so an operator infers a sweep that never runs.
Four doc comments that assert an evictor is called. None is. The only
invocation of DeleteByNode in either repo is inside the /internal/unenroll
executor at shc daemon/wiring/services_extra_formation.go:356, and it
discards the count (if _, err :=), while /internal/unenroll itself has no
client anywhere. DeleteNodeLocal has zero invocations at all.
- ce
modules/backup/ent_ops.go:208-212: “Called by the cluster force-removal path (node rm --force/node forget)”. False. - ce
modules/app/deployment_repo.go:157-161: “Backsshc cluster rm <node> --force… The force path sweeps the deployments first”. False. - ce
modules/cluster/service.go: theDeploymentEvictorandBackupEvictorinterface docs, same claim.
A green test guarding the dead wire. shc
subsystems/rqlite_cluster_seam_merge_test.go:159-162 asserts the evictors
are wired, with a failure message describing a sweep that never happens.
Deleting the claim means re-scoping that test, or the
guard-is-green-while-broken class survives its own fix.
Operator-facing strings that name a verb that does not exist. shc
cmd/shc-x9s/actions.go:398 renders shc cluster forget <node> -y, and
cmd/shc-x9s/catalog.go:633 documents “evict the node from the cluster (shc
cluster forget)”. The verb is shc node forget. The same block asserts at
actions.go:393-396 that “the daemon’s forget route requires the
confirmation flag … Sending yes:false would fail every time”, also false:
handleClusterForget decodes ForgetRequest.Yes (lifecycle_router.go:96-99)
and never reads it, so the confirmation the provider and the HUD send is
decorative. If forget gains a gate, that confirmation has to become real
server-side.
Two refusals whose text needs work. X400017NODDEP currently reads “node
has stacks, rerun with force=true to override” (shc
modules/cluster/admin_router.go:177-179); it names an API parameter, not
anything an operator types; it should name shc node drain <node> and the
CLI flag. And X409203CLSRMM, the sole-swarm-manager refusal, deliberately
names no remedy, with an unusually careful comment explaining that SHC has
none and that naming shc cluster promote was considered and rejected
because it does not exist (ce modules/cluster/errors_extra.go:499-510).
That reasoning is sound and should stand, with one narrow amendment: when the
node’s state is already unreachable, a remedy DOES exist, because
shc cluster repair runs only EnsureVoterRemovable and no swarm guard
(shc modules/cluster/admin_router.go:85-101, read in full), so it reaps a
dead sole manager that forget refuses. Naming it conditionally on that state
keeps the discipline; naming it unconditionally would break it.
What it costs
Section titled “What it costs”Schema: nothing. Migration: none. DDL mirror: untouched. This is the
recommendation’s main advantage over a stored STRANDED state, and it is worth
stating precisely, because the asymmetry is real and easy to miss.
deployment.status carries a MATERIALIZED sixteen-value CHECK containing
neither stranded nor paused
(migrations/20260727022830_auto.sql:292), on the table targeted by six
inbound foreign keys, so adding a value means a SQLite rebuild of the most
FK-encumbered table in the schema, under append-only make migrate on two
live estates, with the hand-written shc DDL mirror moving in lockstep.
node.cluster_state has NO materialized CHECK (grep -c "cluster_state IN"
over that migration returns 0) and already carries removed, written by the
removal itself. Stranded-ness is therefore a JOIN predicate, not a column.
Route by route:
| change | cluster rm | node forget | cluster repair |
|---|---|---|---|
Accept removed at presumed-dead GATE 1a | ✓ | ✓ | ✓ |
| Delete the four counters and their renders | ✓ | ✓ | n/a |
Occupancy gate + --force | ships already | new | not gated (correct: it reaps confirmed-dead members) |
| DNS keep-set predicate | ✓ | ✓ | ✓ |
shc ls stranded annotation | ✓ | ✓ | ✓ |
| Fix the x9s wrong verb | n/a | ✓ | n/a |
Two predicates that are not optional. Retention is only defensible if the retained row stops lying, and both consequences are one WHERE clause each:
- DNS.
ListIngressIntentsisFROM deployment INNER JOIN host_mappingwith no node-state predicate and no filter of any kind (modules/app/deployment_repo.go:328-350, SQL read verbatim), and it builds the DESIRED set for the orphan reaper, the Traefik file provider and the apps menu. A retained row keeps its hostname in DESIRED forever, so an A record aimed at a cloud IP the provider has already recycled is never released. Retention CAUSES that, so the predicate ships in the same change or the ruling is net-negative. The query already SELECTsdeployment.node_id, so the fix is a join, not a subsystem. - Status truth.
repoRegistry.Listcallsrepo.List(ctx, false)(modules/cluster/register.go:65-67) →ClusterStateNEQ("removed")(modules/node/repository.go:128-134), so the health poller can never reach a removed node andMarkContainersUnknownnever fires for it. Rows on a force-removed HEALTHY node therefore readrunningforever. Cheapest correct form is display-only annotation from the node row.
And quota leaks either way, which is worth naming as a known cost rather
than pretending retention is free: recordQuotaDelete runs only in
Service.Uninstall, and the usage table has no FK to deployment, so
retained rows stay counted against tenant quota, while deleting them via
DeleteByNode (a bare Delete().Where(NodeID(...))) would leak the counters
in the other direction.
How this interacts with the binding model
Section titled “How this interacts with the binding model”Derived bound_to. Compatible, and this analysis strengthens the case for
deriving rather than storing. Nothing here needs deployment.node_id to be
correct, because nothing here computes a per-node deployment SET in order to
destroy it; the occupancy gate only needs to know whether the count is
non-zero, and X400017NODDEP already reports it that way. That matters
because the set genuinely is not computable on swarm today: stampNode
records the node that HANDLED THE INSTALL (modules/app/store.go:160-172),
so any destructive verb keyed on it would be wrong in both directions. A
recommendation that never keys destruction on that column is unaffected by the
defect the binding model exists to fix, and can land before it.
STRANDED in slice 5. Affected in scope, not in direction. The state this
analysis needs is exactly slice 5’s, and the recommendation is that it be
built as a DERIVED predicate over node.cluster_state = 'removed' rather than
a stored deployment.status value, for the CHECK-constraint asymmetry above.
The counter-argument deserves recording, because it is the strongest one
against this section: a derived predicate holds no state of its own, so a
stranded row silently un-strands on a same-name, same-IP rejoin:
resolveEnrollIdentity adopts the old node id when the joiner’s physical IP
matches (shc modules/cluster/enroll_handler.go:751-776), consulting
cluster_state in neither branch, and nothing verifies the disk was not
reimaged in between. If that transition cannot be gated or at least made
loud, the stored form earns its migration. Slice 5 should also note that its
dead-node path does not function at all until the GATE 1a fix lands.
Q2: is the DATA-LOSS refusal --force-able? This analysis does not
answer it, but it narrows what the answer has to cover, in two ways. First,
--force on the removal verbs is currently AMNESIA, not discard: it bypasses
an occupancy gate and evicts a raft member, and it destroys nothing. So
ruling on Q2 is choosing what --force should mean, not amending what it
means. Second, the recommendation above deliberately keeps the removal-path
--force non-destructive precisely so that Q2’s ruling has somewhere clean to
land: if the answer is “non---force-able with --discard-local as the only
door”, that door already exists on shc move --presumed-dead
(modules/clone/commands/main.go:272-273) and the removal path does not
compete with it. If a discard on removal is ever wanted, the classifier
DeleteNodeLocalBackupRows already contains (location is null, empty, under
<stateHome>, or restic:<stateHome>/…
(modules/backup/ent_ops.go:224-259)) can distinguish “every row here
already has an off-node backup” from “this is the only copy”, which is the
distinction Q2 turns on.
Q8: should cluster repair’s 24-hour auto-eviction survive? The premise
needs correcting before the question can be answered; see below.
Corrections to the record
Section titled “Corrections to the record”These contradict claims that are load-bearing elsewhere, including two in this document.
- This document’s Q8 says
shc cluster repair“deletes deployment rows and node-local backup rows”. It deletes neither. Repair callsNodeRemover.Remove(shcmodules/cluster/admin_router.go:101), which touches no deployment row and no backup row.DeleteByNodeandDeleteNodeLocalhave zero production callers in either repo. Q8’s engineering premise is false, though its product instinct (that a time-triggered destructive sweep is the wrong shape) is exactly why this analysis recommends against making it true. - The 24-hour leg of repair is unreachable dead code. It reads
m.LastSeenAt(admin_router.go:61), andlast_seen_atappears in NO migration and in no ent schema:git grep -c last_seen_atovermigrations/at the pin returns nothing, andent/schema/node.gohas no such field.LastSeenAtsurvives only as a Go/wire projection whose own comment says no production constructor populates it (modules/cluster/types.go:90-96). The sole live selector ism.State == "unreachable". So Q8 should be re-asked as “should repair’sunreachablesweep survive”, and its real hazard is different from the one stated: the tidy path takes no confirmation at all (-yis read only inside theif forcebranch), is not dry-run by default, and fires on a roughly six-second signal. - Repair is not unattended. No cron, no scheduled job, no CI caller; it is registered as a CLI verb and invoked by an operator. Descriptions of it as an automatic sweep, including in the brief this analysis answers, overstate it.
- The counters print nothing false. Every render is guarded and every
JSON field is
omitempty(admin_fleet.go:279-287). Framings of this as an operator being shown a fabricated eviction count (including in two of the three cases put to the panel) are wrong about the mechanism, though right that the fields must go. - Line numbers drift between the local ce checkout and the pin.
NodeRemovalResult’s fields are atmodules/cluster/service.go:2683-2689at182e4b832582, not the2474-2478cited in two cases. Anyone verifying this section should usegit show 182e4b832582:<path>; the local ce working tree is an ancestor andEnsureSwarmManagerRemovablein particular exists only at the pin. - The
backup.repodefault does not flow through the config layer. One lane reported it as a config-resolved default; it is not. The config surface returns “key not found” on both estates while the repo plainly exists on disk, because the value is set at boot fromos.Getenvbybackup_mount(subsystems/impl/backup_mount.go:416-425). The conclusion is unaffected; the mechanism is observably different and matters to anyone trying to inspect or override it.
The weakest point in this recommendation
Section titled “The weakest point in this recommendation”That a retained deployment row is worth enough to justify the drift it
causes. The recommendation’s positive case is “keep the record, it is the
only inventory the estate has left”, but the record may be insufficient for
the thing it is being kept FOR, which is rebuilding the stack. The row
carries app name and version, runtime, storage, providers, access and
exposure; host_mapping carries the published hostnames. The install-time
var_overrides and host specs do NOT live in either; they live in the
manifest sidecar on the node’s disk (lane 2 traced a live example carrying an
actual secret value, mode 0600). The backup row’s surviving manifest blob
would carry them, but per that same lane no production stack on either estate
has a backup. So if the retained row turns out to be diagnostics rather than a
rebuild input, what this ruling preserves is worth less than it prices, and
the DNS, quota and route drift it accepts gets relatively more expensive,
which is the strongest form of the argument for deletion.
What would overturn it: take one production stack, delete nothing, and
attempt a from-scratch reinstall using only the DB row plus host_mapping. If
the result is not equivalent, the recommendation still holds on
recoverability grounds (nothing was destroyed and the bytes remain on the
disk while the VM lives), but its positive case collapses to “retention costs
nothing”, and the honest headline becomes that the real asset is an off-node
backup neither estate has, which no ruling about removal semantics
substitutes for.
A second, cheaper check should run before implementation rather than after.
The claim “retention cannot lose a byte” requires that nothing ELSE reacts
destructively to a cluster_state = 'removed' stamp. A grep for that constant
at the pin finds only the registry list filter and the certificate rotation
path (which refuses to renew a decommissioned node’s identity
(modules/cluster/certs_internal_router.go:208-222), a deliberate revocation
and not a data action) plus the writers themselves. That is reassuring but
not exhaustive: volume detach and fencing, fabric_pin and
fabric_allocation (both FK-less, and the same-IP adoption branch reuses the
old node id, which is exactly where a stale ordinal could collide), and the
backup lease reaper were not each read in full. If any of them tears down a
mount or reaps an archive on seeing the tombstone, the floor this
recommendation stands on is not a floor.
References
Section titled “References”- 2026-08-02-node-addressing-model.md
the sibling design this one consumes: its slice 11 is the terraform-boundary
mechanism, its Q11 is refined here, and its glossary owns the word
bind - network-fabric.md: the runtime-neutral cross-node data plane; a future shared data path for ordinary mounts would ride it
- cluster/overview.md: membership and the removal verbs this model gates
- 113-removal-quorum-fix-v2.md: the quorum-floor guard whose non-bypassable posture the data-loss refusal copies
- backup-manifest-in-repo.md: the backup
declaration surface
durabilityextends - operations/disaster-recovery.md where the stranded-deployment runbook belongs once slice 5 lands