Skip to content
SHC Docs

Do we use the saga pattern? And what we should do instead

Status: proposal, awaiting ruling. Nothing in this document changes behaviour. Scope: every multi-step destructive operation in shc + ce. Evidence: three read-only survey rungs (app install/update/uninstall · backup/restore/recover · tenant/cluster/terraform), cite-checked against ce a66b5b3 (2026-08-16 13:54Z) and shc e2eaf657a.

Cite-check note. The three reports disagree on one line number (RollbackFailedInstall’s update early-return: 2653 vs 2691 vs 2694). All three are correct for the tree each read; the working copy at 2fd655b2 is 6003 lines, a66b5b3 is 6109. Every load-bearing claim below was re-read at a66b5b3 and holds. Line numbers here are a66b5b3.


No, we do not use the saga pattern, with one exception, and it is excellent.

clone.Move’s heavy path is a real saga: a write-ahead journal that is fail-closed (move_journal.go), two independent evidence gates before the destroy (health, then bytes), a compensation that quarantines rather than deletes, a daemon-start resume, and a refusal (never a guess) once the source has been torn down. It is one verb out of roughly thirty.

What the rest of the system actually guarantees today, by class:

classoperationsthe guarantee
additivebackup create, CloneSafe by nature. A failure leaks bytes, never destroys them.
converge-on-retryinstallReal. Re-running the same install over a live row converges; SettleDeployStatus stamps the verdict from an engine probe, not from the absence of an error.
destructive-in-placeupdate, restore, rollback, group restore, recover, clone.RenameNone. No compensation, no checkpoint, no resume. The destructive act (compose up recreate; removeAllBestEffort per mount) happens before the failure is knowable.
one-way doorsuninstall, tenant delete, tf node destroy, presumed-dead moveA pre-commit refusal, implemented anywhere from excellent (presumed-dead move’s two gates; node destroy’s occupancy refusal) to decorative (the uninstall backup gate) to absent (tenant delete; daemonless uninstall).

Two disciplines the tree already has right, stated once so the rest reads as signal: completion on evidence (SettleDeployStatus, seal.go:verifySealedMember, VerifyRestoreSourceInventory), and node-local crash journals for the backup quiesce bracket plus lease_boot.go’s boot reconcile. Every one of these exists in exactly one module. The defect is not absence of the idea. It is that no rule is applied uniformly.


2. Reachable half-finished states, ranked by what a customer loses

Section titled “2. Reachable half-finished states, ranked by what a customer loses”

Data loss beats a stuck row beats a leaked byte.

#statehow it is reachedwhat is lost
1shc uninstall with the daemon down destroys volumes with no archive and no warning. GateUninstallBackup returns nil on its first line when preUninstallBackup == nil, and SetPreUninstallBackup is called only from daemon/, never from cmd/shc. RunCLIUninstall then runs ExecuteComposeDown{RemoveData: true} (modules/app/register.go:533-553). The adapter’s own comment concedes it: “nil in the daemonless CLI … so this is a no-op there”.“the daemon is wedged, let me clean this up”All of one deployment’s data. Silently. Permanently.
2Daemon uninstall proceeds on an archive that does not exist. The gate keys on err != nil and never reads the returned row; phases.go could return (completed, Location: "").master-archive assembly fails; teardown continuesSame as #1, with a green row claiming otherwise. ce!22 closes the producer half.
3Recover leaves a live, healthy-looking, EMPTY stack. install succeeded, restore failed, postgres initdb’d a fresh cluster, nothing tore it down and nothing marks it.DR replay with one failed memberThe entire point of DR, discovered late, by a user.
4Restore stranded mid-wipe. The inventory proof is whole-archive; the wipe is per-mount (removeAllBestEffort). Group restore takes no per-member safety backup at all, and its doc comment says “atomically restores”.any failure after mount 1 of NPartial, unrecoverable data, unless a pre-restore backup exists and has a location.
5clone.Rename half-rename, permanently. Dirs and volumes at target, DB row at source, stack down. Retry 404s on GetStack(source). The compensation (restartSourceBestEffort) provably cannot work past step 3; by then the tree has moved.ordinary shc move within one tenant/env/nodeNo bytes destroyed; the deployment is unreachable and needs hand surgery.
6A failed update leaves the stack down. compose up recreates destructively, aborts, leaves the replacement in created; restart: policies govern containers that exit, so nothing acts. RollbackFailedInstall returns immediately on update. Four occurrences on 2026-08-16, four causes, one consequence: the estate’s ingress dark until a human typed docker start.any failed updateAvailability, estate-wide when it is the ingress. ce!20 closes the in-process half.
7Tenant delete silently undone. The tombstone is written at service.go:602, after the row drop at :566 and the scoped sweep at :583. Crash in that window → the realm reconcile finds no tombstone and re-creates the tenant within 60s, on top of a half-swept row set.crash mid-deleteA deliberate delete reverted, with its secrets and connections already gone.
8Rows parked in updating / installing / removing / queued forever. grep over modules/ daemon/ finds only writers and the type table; there is no boot reconciler for any transient stack status. Contrast modules/backup/lease_boot.go, which exists for exactly this.daemon death mid-operationOperator time; a stack that refuses further operations.
9Leaked bytes and estate objects. 26.1 GB across 114 unreferenced files measured on bitspur; orphan Keycloak realms and overlay networks after a failed tenant delete; leaked voter slots and fabric /24s on the unlicensed enroll path; restore.go:1516 leaves a full second copy of every restic restore forever.assortedMoney and capacity.

Read the top four together: they are all durability and verification defects, not compensation defects. That observation is the whole of section 3.


3. The proposal: four layers, not four options

Section titled “3. The proposal: four layers, not four options”

Prescribing sagas for #1–#4 would add an orchestrator and fix nothing. These are layers; each defect needs the cheapest layer that closes it.

Layer 0: verified completion. Never record success without reading the result back.

Section titled “Layer 0: verified completion. Never record success without reading the result back.”

Cost: near zero. Runs on every path, therefore constantly exercised, therefore trustworthy. Closes #2, most of #3, and the whole “optimistic completion” class. The rule: a command’s exit status is a claim; the row must carry a fact. The tree already does this correctly three times (SettleDeployStatus, verifySealedMember, VerifyRestoreSourceInventory) and incorrectly in the places that bit us. This is the answer to most of the owner’s question.

Cost: one if and an error code. Fails safe by construction. A wrong refusal costs an operator a flag; a wrong unwind costs data. Closes #1, hardens #2, #3, #7. The rule to adopt verbatim, because move’s data gate already states it: “I could not look” must never share an answer with “it is fine.” Currently violated by GateUninstallBackup (nil hook → nil), by node_join_recovery.recoverAlreadyClustered (membership unreadable → proceed with the destructive --force wipe anyway), and by rollback.go:69 (err == nil discards a failed checkpoint and continues into a replace-restore).

Layer 2: write-ahead intent + idempotent resume.

Section titled “Layer 2: write-ahead intent + idempotent resume.”

Cost: a durable record per operation, a boot reconciler, and idempotency on each step. Closes #5, #6’s crash half, #7, #8. Its virtue over compensation is decisive: if the steps are idempotent, the resume path is the same code as the normal path, so it is exercised by ordinary use rather than only by disaster. Two correct implementations already exist to copy: move_journal.go (write-ahead, fail-closed, “over-record is free, under-record is the bug”) and lease_boot.go (boot reconcile on node-identity proof, 6.82s vs the old 16m). The two journals in the tree disagree on their own core rule: drain_journal.go is write-after and deliberately swallows a failed write. Pick one rule (move’s) and make the other conform.

Layer 3: full saga: compensation per step, orchestrator, reverse unwind.

Section titled “Layer 3: full saga: compensation per step, orchestrator, reverse unwind.”

Recommend against as a general pattern. Two reasons, both mechanical rather than aesthetic:

  1. The underlying tools have no inverse. docker compose up recreates destructively and offers no rollback; removeAllBestEffort unlinks. “Compensating” the destructive-in-place class would mean restore-from-backup, a second destructive operation with its own failure modes, run on the rarest path in the product. That is a worse trade than refusing.
  2. Compensations are the least-tested code we own, and this tree proves it (section 4).

Where compensation is right, it is narrow and forward-only: docker start on what a converge stopped (ce!20), ForgetSnapshot on an unverifiable seal, quarantine-instead-of-delete in rollbackFailedMove. Adopt that form. Do not build an orchestrator.

Only two operations are genuinely saga-shaped: recover (#3) and group restore (#4), where step N’s failure really does invalidate step N−1 across members. Even there the right compensation is mark and refuse, not unwind: an empty recovered stack should be stamped unusable and kept out of the wiring phase, not auto-uninstalled. Auto-uninstall is a destructive act triggered by an error path we cannot test.


4. What it costs, and where “refuse and require a human” is the honest answer

Section titled “4. What it costs, and where “refuse and require a human” is the honest answer”

A compensation runs only when its forward step failed. That is why every compensation in this tree that is not routinely exercised is wrong, and we can show it rather than assert it:

  • Rename’s restartSourceBestEffort cannot work after step 3; it restarts a name the registry no longer holds, or a tree that has moved. No test caught it; nothing exercises the path.
  • RollbackFailedInstall’s update early-return rests on a premise that is false for the failures that actually occur (“the stack is still live”). Four measured incidents disproved it in one day.
  • Against those: move’s rollbackFailedMove is correct, and the reason is structural: ResumeInterruptedMoves runs it at every daemon boot and keeps failed rollback records for retry. The compensation sits on a hot path by construction.

The rule this implies: a compensation you cannot arrange to run routinely must be replaced by a refusal. Three acceptable exercise mechanisms, best first:

  1. Make it the resume path: the boot reconciler runs it on ordinary restarts (move).
  2. Make it forward-only and idempotent, so it is safe to run twice and can be an operator verb (docker start, ForgetSnapshot).
  3. Fault-injection tests that kill between named steps. Expensive; reserve it for the two saga-shaped operations only.

Operations where the answer is refuse and require a human, and we should say so in the product:

  • uninstall: no inverse exists. Verify the archive; refuse without one.
  • cluster leave past storage detach: the code already says a refusal here cannot be undone.
  • presumed-dead move: rebinding to a dead node is never correct. Already refuses; correct.
  • recover with a failed restore: mark unusable, refuse to wire it in. Do not auto-uninstall.
  • move past SourceTornDown: already refuses, and correctly. But the --finish-forward verb the refusal points the operator at does not exist. A refusal with no exit verb is an operator trap; that is a cheap, high-value fix.

5. Migration order, and the ruling on the two changes in flight

Section titled “5. Migration order, and the ruling on the two changes in flight”

Ordered by safety bought per unit of risk taken.

#changelayerrisk
0Merge ce!22 as-is.0low; it makes a green row rarer, never a red one greener
1Merge ce!20 with adjustment A1 (below).0 + narrow compensationlow
2Close the uninstall gate, both halves. (a) daemon: refuse on Location == "" and on DegradedNoDataCaptured, one if. (b) daemonless: preUninstallBackup == nil must be a loud refusal naming --no-backup, not a silent return nil.1~5 lines; the single highest-value change in this report
3One journal rule, one boot reconciler. Make drain_journal write-ahead like move_journal. Add a stack-status reconciler that adopts updating/installing/removing/queued rows at boot and settles them by re-observation; it must not re-run the operation.2medium; closes #8 and #6’s crash half
4Put clone.Rename on the move journal and delete the compensation that cannot work.2medium; closes #5
5Refusals: rollback.go:69 aborts like restore.go:348; node_join_recovery refuses when membership is unreadable; tenant delete writes the tombstone before the row drop; recover marks-and-refuses.1low each
6Leaks: restore.go:1516 defer os.RemoveAll; release-defers on the unlicensed enroll path.n/alow
7Only now, if ever, consider real orchestration for recover + group restore.3high

Is ce!22 (backup read-back verification) consistent with this shape?

Section titled “Is ce!22 (backup read-back verification) consistent with this shape?”

Yes, it is the reference implementation of Layer 0. Merge unchanged. Three things it gets right that the rest of the estate should copy:

  • It fixes the producer, not the consumer. One producer, three consumers that currently disagree about what an empty location means. Fixing the gate alone would have left the other two.
  • verifyCapturedSnapshot distinguishes DISPROVEN (forget the snapshot, fail the row) from INTERRUPTED (leave it, say so); that is exactly the Layer-1 rule, applied at Layer 0.
  • The empty-location refusal at finalize (X500473BKPNOA) makes the top of the ranking’s precondition unrepresentable going forward, rather than merely detected.

One follow-up, not a blocker: ce!22 does not subsume migration item 2. It stops new empty-location completed rows; legacy rows written before the merge still exist, the gate still keys on err != nil, and the daemonless hole (#1) is untouched by it entirely. Keep item 2 on the list after ce!22 merges.

Is ce!20 (traefik converge restore) consistent with this shape?

Section titled “Is ce!20 (traefik converge restore) consistent with this shape?”

Yes in shape, with one adjustment I would ask for before merge, and one immediately after.

What it gets right, and should be treated as the template for narrow compensation: it is forward-only (docker start, never a re-converge), idempotent, verified by re-observation rather than by exit code, and it never changes the verdict; the update still reports FAILED. It also correctly leaves RollbackFailedInstall’s update early-return alone: the answer to a failed update is not to tear down a live stack.

  • A1: adjust before merge: the before-image is in-process only. preUp is a local variable in ExecuteCompose (install_runner.go:2223 on the MR branch). The four measured incidents were converges the daemon survived. A daemon killed mid-compose up (which is precisely the failure that already parks rows in updating forever (#8)) leaves the identical dark ingress with nobody left to run the restore. Ask: persist the before-image beside the rendered compose (one small file), so the boot reconciler in migration item 3 can invoke the same RestoreAfterFailedConverge. Without it we ship a fix that covers the observed incidents but not the mechanism, and the follow-up gets much more expensive once the code has shipped in-process-only.
  • A2: land in ce!20 or immediately after: DeriveContainerStatus maps "created" → ContainerStatusStarting (container_status.go:52). So even with ce!20 merged, when the restore itself fails (X500117CVGRSF is a WARN), the health rollup still reports the stack as coming up rather than broken; our own status surface continues to lie about the exact state ce!20 exists to fix. A container in created with no start time is not starting; it is degraded.

Do not adopt the saga pattern. Adopt three rules and apply them uniformly: record completion only from a read-back; refuse at every one-way door, and never let “I could not look” answer the same as “it is fine”; write intent ahead of the act and resume it at boot. Reserve compensation for the narrow, forward-only, routinely-exercised cases we already have. The correct implementation of each of these already exists in this codebase, in exactly one module apiece.