Clone/restore lineage is not rename-aware
Status: open design. Clone and restore still replay the name a stack had when the backup was taken. This page records a live finding from the Roll 2.1 close so the class has a name and a home. The fix belongs to the backup/clone machinery and earns its own lane; nothing in this page changes behaviour.
The finding, measured
Section titled “The finding, measured”bitspur corp/prod, 2026-08-22. The stack now named paperless was renamed
from docs earlier in its life. shc clone paperless failed with
X500603CLNRCV, and the RecoveryOrchestrator’s plan shows why: it tried to
install stack=docs, the pre-rename identity, three times, drawing
X503010NDUNRC (mesh routing), X500960SWMDPL (swarm deploy) and X409006RTMOWN
(runtime ownership: the live paperless swarm objects sit under the project
the resurrected docs render folds onto).
The restore chain carries the OLD name. The archives, their manifests and the recovery plan were written under the identity the stack had at capture time, and nothing in the clone/restore path re-maps that lineage through the rename. A renamed stack’s clone therefore tries to resurrect its former self from stale state, and the failure surfaces as routing/deploy/ownership errors that name everything except the rename.
The adjacent class: import-origin divergence
Section titled “The adjacent class: import-origin divergence”The rename is one member of a wider class: the identity or path a deployment RUNS under today diverges from the identity its durable lineage was recorded under. Known members on the live estates:
docs→paperless(this incident): rename, restore lineage keeps the old stack name.- supabase on the imported estate runs from the
_platformpath, an estate-import artifactshc updatereplays; its lineage is pathed, not named, the way a native install would be. - imported uptime (Uptime Kuma): same import-origin shape, lineage recorded under the origin identity.
Anything that replays recorded lineage (clone, recover, move’s restore leg, possibly scheduled-backup subject resolution) can hit a member of this class. The sidecar-not-where-the-render-looks refusal (X409010DEPNOS / X409011DEPELS) is the same divergence seen from the node axis instead of the name axis.
Shape of a fix (for the future lane, not here)
Section titled “Shape of a fix (for the future lane, not here)”The capture side should record a stable deployment identity that survives rename (or the rename verb should rewrite/annotate lineage), and the replay side should resolve lineage THROUGH the current row rather than trusting the archived name. Fail closed stays the rule: a replay that cannot prove which present-day deployment its lineage names should refuse and say both names, not install the older one.
Until then
Section titled “Until then”A clone/recover of a renamed or imported stack is suspect. If it fails with a plan naming an identity you no longer use, that is this issue; do not retry into the old name; the retries are what stacked three distinct errors on one incident.
Addendum 2026-08-22: the same-day clone-to-new-target failures are FIXED, and they were a different member
Section titled “Addendum 2026-08-22: the same-day clone-to-new-target failures are FIXED, and they were a different member”Root-causing the day’s other two failures (shc clone paperless docs and
shc clone immich photos, both corp/prod, both dying
X500603CLNRCV … restore from file [X404401STKNOF] stack <target> not found after a completed capture) landed on a scope bug, not the rename
lineage above: the clone restore leg’s target scaffold read the SOURCE
sidecar from the scaffolder’s baseline-tenant/default-env fallback, so on
any estate whose deployments live outside that scope it silently no-oped
and the restore resolved a target that was never materialised. That is why
immich → photos failed with no rename anywhere in its history, and why
the shc recover -b <bak>:<target> workaround succeeded (recover installs
its target before restoring).
Fixed in ce (daemon/services_seams_clone.go, ensureCloneRestoreTarget:
read the source from the input’s own scope, fall back to a bare target
scaffold, fail closed by name), ce MR:
https://gitlab.com/bitspur/selfhosted-cloud/ce/-/merge_requests/203,
with the regression pinned in this repo’s corpus
(tests/_ce/daemon/clone_recovery_target_scaffold_test.go).
The capture-side finding this page records (archives and recovery plans carrying a pre-rename identity, and the wider import-origin divergence class) REMAINS OPEN. A clone/recover of a renamed or imported stack is still suspect exactly as described above; what is no longer suspect is a plain clone to a fresh target name in a non-default scope.
Addendum 2026-08-22: the rename’s DNS gap: operator note, and why it stays
Section titled “Addendum 2026-08-22: the rename’s DNS gap: operator note, and why it stays”A renamed host is unresolvable AFTER it is healthy
Section titled “A renamed host is unresolvable AFTER it is healthy”shc uninstall <old> releases the deployment’s managed DNS records and the
shc recover -b <archive>:<new> that follows re-creates them, so the hostname
does not exist for the length of the gap between the two commands. A recursive
resolver that queries inside that window caches the NXDOMAIN for the ZONE’S
NEGATIVE TTL (the SOA minimum, 3600 seconds on bitspur.com) and keeps
answering nothing long after the stack is up and serving. Measured three
times on 2026-08-22 (paperless→docs, immich→photos, huly→projects): the
authoritative NS and 8.8.8.8 answered the new A record correctly while other
recursors (the estate node’s own Hetzner resolvers included) still returned
nothing.
Before calling it an outage, check the two things that skip the cache:
dig +short @<zone NS> <host> # authoritativecurl -sI --resolve <host>:443:<node-ip> https://<host>/ # the node itselfA correct authoritative answer plus a 200 with a valid cert means the stack is fine and you are waiting out somebody else’s negative cache. Nothing on the estate shortens it.
Verdict: the preserving rename already exists, and it is shc move
Section titled “Verdict: the preserving rename already exists, and it is shc move”Investigated whether the rename path could keep the records instead of delete-and-recreate. It can, through the verb the incident did not use.
Where the delete happens. ExecuteComposeDown releases every hostname the
manifest persisted (ce modules/app/install_runner.go:3643-3672), latching the
per-deployment retire tombstone first (RetireDeploymentDNS,
modules/app/dns_retire.go:52-61) so the leader’s proxy_reconcile cannot
re-publish into the window. It runs on EVERY uninstall by design: a record is
per-deployment state, and one left behind orphans in the zone and survives
cluster destroy (2026-07-23 cloud shakedown, daemon bug #2,
install_runner.go:3646-3649).
Why KeepDNS is move-only. A managed record is keyed by HOSTNAME ALONE; a
zone carries no tenant/stack/environment
(modules/app/install_runner.go:4212-4215, modules/clone/types.go:801-808).
Suppressing the release is therefore safe only when another deployment is
ALREADY serving that exact hostname, and the only caller that can prove it is
the move orchestrator: recoverPhase brings
the target up and re-registers the source’s hosts verbatim
(daemon/services_clone_remote.go:484-485) BEFORE uninstallPhase tears the
source down (modules/clone/service.go:798, then :880), which is where
keepDNS=true is passed (modules/clone/service.go:2458) and mapped onto the
teardown (daemon/services_seams_stack.go:807-810, :832). That ordering IS
the invariant. Hand the flag to a bare uninstall and the record dangles over an
address the cloud is free to recycle.
A rename is not distinguishable at the deletion. shc uninstall <old> and
shc recover -b <archive>:<new> are two independent operations with nothing
joining them. The uninstall has no forward knowledge that a recover is coming
and no way to acquire it, so no fix belongs on the uninstall side.
The recover side is already an idempotent upsert, and ownership does not
block it. CreateDeploymentDNSRecords → syncHostRecords →
SyncRecordsForHostByPurpose is a sync, not a create
(modules/app/dns_lifecycle.go:135, :331-342), and it SELF-ADOPTS an unowned
host, minting the ownership TXT and correcting the targets in the same pass
(modules/dns/ownership.go:42-50). Ownership is per-CLUSTER (dns.owner_id)
and hostname-scoped, never per-deployment, so a record this cluster published
under the OLD stack name reads ours under the new one and the upsert is a
no-op decision; a recover onto a still-live record converges cleanly. The
X403049DNSPTO refusals in the same night’s reap are a different axis: a
predecessor claim from another cluster INSTANCE sharing the host slug
(modules/dns/predecessor_gate.go:42-59). It cannot arise inside one cluster’s
rename.
The gap belongs to the workaround, not to the code. shc move <old> <new> is the rename that preserves DNS, and has been since the cross-key path
existed. It was unusable on this estate the night of the incident: a swarm
source took the in-place clone.Rename fast path, which renames nothing on
swarm and reported success anyway (X400635CLNSWR,
modules/clone/errors.go:500-530, measured on the same 2026-08-22 corp/prod
shc move paperless docs). That is fixed in ce main: a swarm source is routed
onto the backup+restore path (modules/clone/service.go:1466-1492), which is
the path that carries keepDNS. clone → uninstall → recover was a workaround
for a broken verb, and the negative-TTL tail is a property of the workaround.
The fix is on main, not on the estates. As of 2026-08-22 the fleet runs a
pre-fix daemon; v0.0.61 was never tagged, so nothing carrying the swarm
routing has been rolled. Until it is, shc move on a swarm source still takes
the fast path, the clone → uninstall → recover sequence remains the only
working rename, and the operator note above is live guidance rather than
history. Re-read this section after the roll; the verb is the answer the moment
the daemon carries it.
No behaviour change taken here. The preservation is implemented and pinned
(tests/_ce/modules/clone/move_teardown_test.go:72, :116;
tests/_ce/modules/app/teardown_dns_release_test.go;
tests/_ce/daemon/services_seams_stack_ownership_test.go). Weakening the
uninstall release to make the workaround cheaper would trade a bounded
negative-cache tail for an unbounded dangling record, which is the wrong
direction. Use shc move; when you cannot, the operator note above is the
answer.
Test the case that breaks, not the case that passes. A rename exercises
vault-secret handling differently depending on how the app declares its
secrets: a ref+vault://key?autogen regenerates silently and sails through,
while an operator-provisioned ref+vault://key fails closed when the
deployment-scoped row is purged. The same app can differ per estate: on
2026-08-22 bitspur’s copy of huly carried an operator-provisioned
huly_s3_secret_key while promanager’s carried
ref+vault://s3_secret_key?autogen&length=40, so the identical rename breaks
on one estate and passes on the other. Any regression test for the
vault-ordering fix must rename a stack whose secret is OPERATOR-PROVISIONED;
one that renames an autogen stack proves nothing and reports a false
all-clear.