This guide is for an estate where backup.repo and backup.password are
already set but shc backup ls shows nothing worth restoring. It gets the
first real archive into object storage, and it names the two traps that make
this operation riskier than it looks.
Read section 4 before you run anything. On a daemon older than v0.0.18 the
first capture of a database stack could leave that database read-only if it
failed: an outage caused by taking a backup rather than by not having one.
v0.0.18 closes that (ce 8111eb3), but the check is thirty seconds and the
remedy is one command, so section 4 keeps both.
v0.0.18 also changes what a capture costs.shc backup create used to read
the volumes with the stack running; ce 729a649 gives it a quiesce bracket, so
every capture now pauses the stack for the duration of the read. Section 3
and the Expected outage table in section 5 are written for that behaviour.
Measured read-only on the leaders (promanager 204.168.129.34, bitspur
37.27.215.210). Both estates are configured identically.
★ THE DEPLOYMENT COUNTS BELOW ARE STALE. Re-derive before you use them.
This section originally recorded an identical 12 deployments on both
estates. They are not identical, and neither number should be trusted
now: the two estates have diverged since 2026-08-04, and bitspur in
particular carries substantially more than 12. Run
Terminal window
shcls--all-tenants
on each leader and work from what it prints. Everything downstream that is
sized by deployment count (the “12 captures” in section 5.4, the “twelve
snapshots” check in section 6, and any estimate of how long the sequence
takes) is only as good as that number.
Section 5’s per-stack sequence enumerates the promanager estate only. It
is a worked example, not a fleet inventory: run it against the output of
shc ls --all-tenants on the estate in front of you, and expect bitspur to
need stacks that are not named there.
5 rows, every one pre-uninstall, every one ENCRYPTION none, newest 2026-07-30
no backups
shc backup schedule ls
no backup schedules
no backup schedules
shc doctor run -c backup
info: key resolves
info: key resolves
Bucket reachability from the leader
HTTP 403 AccessDenied, TLS in 0.11s
HTTP 403 AccessDenied
Live deployments (STALE: re-derive)
12 as of 2026-08-04 (8 in corp/prod, 4 in master/default)
NOT 12: bitspur has grown well past promanager; re-derive with shc ls --all-tenants
Bytes under mounts/
~5.7 GB
~6.1 GB
Free space on /var/lib/shc
556 GB of 601 GB
556 GB of 601 GB
Every one of promanager’s five archives names a stack that no longer exists
(vaultprobe, s3front, s3proof-db, s3proof-nc). Neither estate held a
single archive for any of its live deployments.
shc schedule ls is a different subsystem: per-stack app job crons. The
backup schedule table is shc backup schedule ls. Both are empty; check the
second one when you are asking about backups.
Four things could have caused this. Three were ruled out by measurement.
Configuration missing: no. All four backup.* keys are set at global
scope on both estates.
Vault reference unresolvable, or vault locked: no.shc doctor run -c backup returns the branch that names the repository, and
DiagnoseBackupEncryption only reaches that branch after the config loader has
returned both a repository and a password; a loader error or a locked vault
takes an error: branch instead. For contrast, a daemon with nothing
configured reports [fail] no backup key configured, shc backup create will refuse. Both estates report the configured branch.
Object storage unreachable, or the bucket absent: no. An unauthenticated
GET of the bucket from each leader returns 403 AccessDenied over TLS in
about 0.1 seconds. The same request against a deliberately nonexistent bucket
name returns 404 NoSuchBucket, so the 403 is positive evidence that the
bucket exists and that egress and TLS work.
Nothing refuses. No schedule was ever registered, and shc backup create
was never run. That is the whole cause.
The five promanager archives are not backups anyone took. They are the
pre-uninstall safety archives the orchestrated engine writes automatically when
a stack is removed, which is why each names a deleted stack, and why bitspur,
which has never uninstalled anything, has none.
One vestige worth knowing about: /var/lib/shc/backups/ on both estates
contains an empty restic repository (config, data/, index/, keys/,
snapshots/, created 2026-07-27) with zero snapshots. It is not the
configured repository and nothing writes to it. Leave it alone.
Since 0.1.28 SHC has one backup engine (FZ-7061). shc backup create, each
member of --all-connected / --with-dependants, the backup schedule cron,
the pre-uninstall safety backup, the pre-restore and pre-rollback snapshots,
clone and move all run the same capture: freeze, copy, .bak, then FINALIZE
seals that .bak into backup.repo, reads it back from the repository at its
byte size, and deletes the local copy. The row reads encryption=restic and
location=restic:<repo>:<snapshot>.
every backup
Encrypted
Yes, by sealing. A scope with no backup.repo or no backup.password is refused before anything is captured (X400486BKPNRP; a scheduled fire is skipped with X400079BKPSKD)
Destination
backup.repo (your bucket), redirectable at every config scope
A seal that fails
fails the run (X500480BKPNSL; scheduled X500474BKPSRQ) and discards the plaintext with the run’s staging. Nothing rests on local disk
Stack outage
pause, escalating to a full stop past timeouts.quiesce_hold (300s); the retake is a delta over the copy already made. --fail aborts at the hold instead, with the stack running
shc uninstall on a scope with no repository therefore stops at its safety
backup: configure one, or pass --no-backup to tear down without an archive.
A restore that takes a rollback point stops the same way (--no-rollback).
Archives shc backup create wrote up to 0.1.27 are restic tree snapshots of
the deployment directory; they stay restorable, exportable and verifiable.
What the key does not cover. Three artifacts stay plaintext on purpose, and
each is named on every branch of the backup.encryption doctor probe:
Artifact
Written by
Why it is not encrypted
{state_home}/exports/<id>.bkg
backup create --all-connected / --with-dependants, backup group
It bundles members by DECRYPTING them back out, because it exists to be read by shc recover --file on a cluster with a different key. Owner-only (0600 in a 0700 dir) and never pruned by retention; delete it yourself. For clusters that share a backup.password, shc recover --consistency-id <id> reads the same group straight out of the repository and produces no plaintext at all.
pre-move archive
clone/move
A kept pre-move archive rests into the stack’s backups dir as an ordinary backup, labelled as pre-move, and from then on the normal retention applies to it. The capture stages under <staging_dir>/clone/<operation id> and that directory is removed when the operation ends.
backup create --file / backup export output
you
You asked for a portable archive at a path you named. A --file create writes only that file: it records no backup and takes no repository snapshot, with a repository or without one. The archive is plaintext on disk from the start of the capture: it grows at <staging_dir>/export/<id>.bak (0600 in a 0700 dir) while the capture runs, then waits there on the node that staged it until the CLI takes it, and the take unlinks it. One the CLI never takes (an interrupted or timed-out create) is removed an hour after it was staged by backup_recovery_sweep, the node-local sweep that runs every minute on every node. One that never finishes staging is removed once it has been staging longer than timeouts.backup_stack (6h unless you set it), the budget every capture runs under. The first sweep after a daemon restart removes whatever the previous daemon left there. Keep SHC_STAGING_DIR on each node’s own disk (one staging dir per node).
Every new backup is encrypted, or it does not exist. The seal is not
conditional any more: no repository, no key, an unreachable repository, or a
seal whose in-repo listing cannot be verified all fail the run with the reason
named, and the seal waits up to five minutes for a lock another restic
operation holds on the repository, such as a prune’s, before it gives up. A
schedule created while backup.repo is set and backup.password is not is
refused outright (X400077BKPSCD).
Plaintext archives from before 0.1.28 may still stand in
{state_home}/backups. An unsealed orchestrated .bak is a plain !<arch>
container holding manifest.json plus an uncompressed tar, with no key needed
to read either. On a real app it carries app/vars.yaml, the deployment’s
install-time --var values, in the clear, and the app’s own credential files
verbatim: a Nextcloud config.phpdbpassword, a frappe site_config.jsonencryption_key, a postgres converter’s pg_dumpall role lines. The one
sealed member is secrets.vlt. Treat {state_home}/backups as secret
material, and prune what is already off-site.
ce modules/backup/artifact_bytes_test.go plants a credential-shaped canary in
the payload, runs each orchestrated caller, and reads every byte of every
artifact root looking for it, with a paired positive control that requires the
canary in a caller-named --file archive, so the absence assertions cannot
pass over a capture that captured nothing. A second tier drives a real
restic binary against a real repository and searches the repository’s own
bytes.
This records how the restic-direct engine, removed in 0.1.28, came to freeze.
Both engines now freeze. Until v0.0.18 the encrypting engine took no pause
and no stop: CreateBackup held no quiesce bracket at all, only app-level
pre_backup hooks, confirmed by polling docker inspect 300 times across a
capture with zero paused samples. The engine that protected the data did not
encrypt it and the engine that encrypted it did not protect it, and
shc backup ls rendered both identically.
ce 729a649 closes that half. CreateBackup now holds quiesce.go bracketPause (the same primitive the orchestrated flow and the routed
single-mount RPC already shared, not a second copy), in the order
Three properties are worth knowing before you run it:
The release is armed by defer in the same statement group that acquires
the freeze, so a staging error, a restic failure, a cancelled request or a
panic all still unpause the stack.
Only the read is inside the window. The whole manifest is built with the
stack live; under the freeze SHC stamps the consistency claim, writes the
few-KB manifest and runs restic.
A failed freeze does not abort the capture. It stamps the archive live +
not_quiesced and proceeds, so a stack SHC cannot pause still gets a backup
and the row says which kind you got. Whether the restic path should instead
escalate to a full stop the way pause_hold.go does is an open owner call.
So a per-stack shc backup create is now encrypted and a real point in
time, at the cost of a pause it did not used to take. Each stack is still its
own point in time. --all-connected gives a true consistency group and is the
only whole-product restore for affine, huly and keygen (whose own
backup.yaml files say so), but it is plaintext and on-box. Section 5 takes
the encrypted per-stack route and section 5.4 explains how to order it so that
skew degrades gracefully.
(--all-connected needs a non-empty integration graph. On a stack with no
connections it refuses with X400403BKPFAL … at least one path is required,
which is expected rather than a fault.)
An app bound to a shared postgres, mysql or mariadb engine keeps its rows on
that engine, not in its own volumes. Its per-stack backup now carries them
anyway: the backup dumps the one database the app’s plug names through the
engine’s exporter, and no other database of that engine. shc backup ls shows
each dump and its size under database dumps. A lone restore imports the dump
onto the engine the app is bound to, creating the database if the engine lacks
it, and leaves every other database alone. A failed dump fails the backup
with X500848BKPDBD. The engine’s own backup is unchanged, and an
--all-connected run that captures the engine does not dump the app’s database
a second time. shc backup restore --paths database/<plug> restores the
database alone.
4. The trap: a failed capture could leave a database read-only
FIXED IN v0.0.18 (ce 8111eb3). If your daemon is older, this is live.
The four postgres stacks per estate declare a hook pair in app.yaml:
pre_backup: [quiesce] # ALTER SYSTEM SET default_transaction_read_only = on; CHECKPOINT
post_backup: [resume] # ALTER SYSTEM SET default_transaction_read_only = off
That is how a capture stays safe for a database: postgres goes read-only and
checkpoints, so the filesystem read is consistent. The quiesce is
persistent: postgres writes the switch into postgresql.auto.conf, so it
outlives a container restart and a reboot. The only thing that takes it back off
is the post_backupresume hook. Every database SHC packages does something
in this family: supabase the same ALTER SYSTEM, mysqlSET GLOBAL super_read_only = ON, mariadbSET GLOBAL read_only = ON,
nextcloud maintenance mode.
Note the two mysql-family spellings are not interchangeable, and this page said
they were until 2026-08-08. super_read_only is a MySQL 5.7+ / Percona
variable; MariaDB has never implemented it (MDEV-9458 is closed WON’T FIX),
and a mariadb server answers ERROR 1193 (HY000): Unknown system variable.
Their persistence also differs from postgres’s: SET GLOBAL is not written to
disk, so a mysql-family gate that the resume hook fails to clear survives
until the container next restarts: long enough to matter, not forever.
What was wrong before v0.0.18: the undo was an ordinary statement on the
happy path. On the restic path runBackupPostJobs was called on the success
path only, with eight early returns between it and the pre_backup call;
on the orchestrated path it was the last statement of FINALIZE’s success branch,
so every failure return and the whole abort() path skipped it too. One of the
eight is EnsureInitialized, the first-ever restic init against the bucket,
the single most likely thing to fail on a first run, and it sat inside the
read-only window. So an operator’s first backup attempt, failing on a typo’d
S3 key, a wrong region or blocked egress, armed the read-only switch and then
left. Measured on a disposable daemon against the pre-fix tree: INSERT failed
with cannot execute INSERT in a read-only transaction, and a container restart
did not clear it.
How v0.0.18 closes it: the undo is bound to the thing that armed it rather
than added at another call site. openBackupBracket fires pre_backup and
returns a closer the caller defers at the arming site, so every exit path
(early return, error, cancellation, panic) pays it. It is once-only (the
happy-path call keeps its original ordering and the hook never double-fires), a
no-op when nothing was armed, and its context deliberately drops
cancellation: half the exits it backstops are a cancelled or
deadline-exceeded request, and an undo bound to the caller’s context would
decline to run in exactly the cases that need it most.
The undo’s own failure is now reportable, which it was not: the seam was
void, so the backup module had no way to say a resume hook had failed. It
surfaces as X500459BKPUNQ (500, “backup of <stack> could not un-quiesce
the stack, it may still be read-only”). It never replaces the capture error
that got you there. If you see that code, run the check below: it is the
one remaining path to a stuck read-only database, and it is now loud instead of
silent.
4a. A quiesce is a HARD ERROR to your applications, not latency
Whatever the daemon does with the outage (a pause, or the full stop it
escalates to past timeouts.quiesce_hold), the app-level pre_backup quiesce
that runs before it does not make clients wait. It makes them fail.
ALTER SYSTEM SET default_transaction_read_only = on does not queue writes.
A connection that is already open and takes a write lock during the window gets
an error back:
ERROR: cannot execute SELECT FOR UPDATE in a read-only transaction
ERROR: cannot execute INSERT in a read-only transaction
The read-only gates on mysql/mariadb and nextcloud’s maintenance mode behave the
same way in their own vocabulary. So a scheduled backup of a database that
other stacks talk to is, for its duration, an outage those stacks see as
5xx, not a slow patch, and an application whose retry policy assumes
transient latency will surface it to users rather than ride it out.
Two things follow for how you schedule and monitor:
Schedule database captures in a window you would accept a short failure in,
and treat a health check over a dependent app the same way you would around
a restart, not around a slow request.
If an app must ride a backup without erroring, it needs a retry that treats
read-only transaction as retryable, or the stack needs to be captured
through a path that does not arm the app-level quiesce at all
(leaving a mount out of include in the stack’s backup.yaml takes it out
of the capture; it does not remove the hook).
This is a property of the quiesce mechanism, not a defect in it: a read taken
over a database that is still accepting writes is the untrusted archive the
whole bracket exists to prevent. It is documented here because the failure is
reported to the application, which means it lands in the app’s logs and not
in shc’s, so an operator correlating a 5xx spike with a backup window has no
other place to learn the connection.
Three consequences for the procedure. The first two are cheap insurance rather
than strict necessity at v0.0.18, and remain mandatory on an older daemon:
Prove the object-storage path on a stack with nopre_backup hook first.
Only openobserve and the four postgres stacks have hooks, and only
the postgres pair leaves sticky state.
Capture the databases one at a time, checking each afterwards.
Know the remedy before you need it.
Terminal window
# check (safe to run any time)
sudodockerexecmaster_default_postgres-postgres-1\
psql-Upostgres-tAc"SHOW default_transaction_read_only"# want: off
# remedy if it says "on"
shcjobrunresume-tmaster-spostgres# or:
shcjobrunclear-read-only-tmaster-spostgres
Baseline on 2026-08-04: all eight postgres instances across both estates
reported off. Nothing is currently stuck.
Run these on the leader. Substitute the tenant and environment shown: the
corp deployments live in environment prod, the master ones in default.
--fail tells a capture to abort rather than escalate to stopping a stack. As
of v0.0.18 the encrypting path holds a pause of its own, so this is no longer
merely insurance against landing on the orchestrated path by accident; keep it
on. Note the asymmetry: the restic path does not escalate a failed freeze to
a stop at all (it stamps not_quiesced and proceeds), so --fail bites on the
orchestrated path and on the hold deadline, not on a pause SHC simply could not
take.
keycloak has no mounts/ directory and no pre_backup hook, so this is the
cheapest possible probe of credentials, restic init and the first snapshot. It
produces a small metadata-only archive; that is expected and still proves the
path. (Verified on a throwaway stack with no volumes: the row came back
encryption=restic.)
Keycloak keeps no data of its own here: its realms live in the keycloak
database inside the master/postgres stack (KC_DB_URL points at
default.postgres.postgres:5432/keycloak), so section 5.4 is what actually
protects your realms and clients.
Capture each database before the stacks that reference it. With independent
snapshots, a database older than its blob store yields orphan blobs, which are
harmless; a database newer than its blob store yields dangling references, which
are broken attachments. Older-database is the direction that degrades
gracefully.
Each database carries the read-only hook from section 4, so do these one at a
time and check the flag cleared before moving on.
Each app’s own capture below also dumps its database from the engine (section
3), so an app restored alone gets its rows back. The engine captures stay the
copy of the whole engine.
huly keeps its rows in huly-db and its uploaded files in the hosted
provider’s bucket, reached through an external-s3 front. Its own volumes
hold the derived fulltext index and queue offsets. Back up the hosted bucket
separately after the database capture. See
Object storage for apps for the connection setup.
affine and keygen keep their blobs in their own volumes, so each is a
plain pair.
Terminal window
# --- keycloak's database (keycloak itself was captured in 5.2) ---
This is not a fleet inventory. Reconcile this list against
shc ls --all-tenants on the estate you are working on, and add a capture for
every deployment it names that does not appear above. Bitspur in particular
needs more than these.
Exit code 0 is not verification. This project has shipped a truncated dump that
returned 0, and there are two specific ways a backup can look healthy here and
not be.
shc backup ls showing a completed row is not proof the archive exists.
Demonstrated: after deleting a snapshot from the repository behind the daemon’s
back, shc backup ls still listed the row as completed with
encryption=restic. Only SIZE BYTES dropped to 0. shc backup sync did not
reap it either. Do not treat the list as evidence.
Use these instead, in order of strength.
Check the posture per row.encryption must be restic and location must
start with restic:. A row reading none came from the plaintext engine and is
sitting on the same disk as the data.
Terminal window
shcbackupls--all-tenants
Confirm the snapshot is present in the repository. This asks the
repository, not the index, and it genuinely fails when the snapshot is gone,
verified by deleting one and watching it return X400405BKPINV while an intact
sibling still passed.
Terminal window
shcbackupverify<backup-id>
Count the objects in the bucket. Ground truth, independent of SHC. Run on
the leader; the credentials are the same backup.aws_* values the daemon uses.
resticsnapshots# expect one snapshot per stack you captured
resticstatslatest# non-zero size
resticcheck# repository integrity; verify only probes membership
You should get one snapshot per stack you captured; compare against your own
count from shc ls --all-tenants, not against this guide’s twelve. Fewer means
a capture failed quietly.
Restore-test one stack. The only check that proves the bytes are usable.
Pick something you can afford to stop, and note the stack is stopped for the
replay. Measured on a small stack: 1.79 seconds stopped, plus 364 ms paused
while the pre-restore rollback archive was taken. Real stacks scale with the
data being replayed, not with the archive size.
Terminal window
shcbackuprestore<backup-id>-s<stack>--yes
Then compare a checksum inside the container against the value from before the
restore. On the throwaway proof both a text sentinel and a 64 KiB binary blob
came back with identical MD5 sums after being deliberately corrupted, so the
replay is real rather than a no-op.
A restore takes its own pre-restore rollback archive first, and that one is
written by the plaintext engine to local disk. Expect a new encryption=none
row to appear; that is the mechanism, not a fault.
An uninstall takes a pre-uninstall safety backup, then drops the
deployment’s config and vault secrets. A database or bucket it held on a
provider stays there unless its plug’s cleanup was set, and --no-backup
keeps its block volumes. That leftover data guards against an accidental
loss. It is not where a later install resumes: an install under the same name
is a new deployment, and it refuses while any of it is there, naming each
leftover (X409440LEFTOV, X409440VOLKPL). shc install --from-backup is a
new deployment as well and refuses the same way.
The way back is shc recover with that safety backup. It brings the
deployment back over the leftovers, with its data and its vault secrets:
Terminal window
shcbackupls# the stack's newest pre-uninstall row; LOCATION is the archive
To start fresh instead, delete the leftovers the refusal names, then install.
Set --var <plug>.cleanup=true, on the install or a later shc update, and
the uninstall drops them itself.
Everything above is a one-off an operator has to remember to run. Ruling B3
(docs/decisions/2026-08-14-backup-rulings.md) makes the deep tier standing:
the package ships /usr/lib/shc/scripts/backup-verify-sweep.sh plus a
shc-backup-verify.timer/.service pair that runs a weekly, budget-capped
shc backup verify --deep of the newest good archive per deployment:
newest completed row whose location is a restic snapshot. Deployments are
visited stalest-turn-first (then smallest-first): a turn is the last time the
sweep started a deployment’s read, proven or cut short, kept in
/var/lib/shc/backup-verify/last-attempted.tsv beside the proofs in
last-proven.tsv. A budget that covers only part of the estate rotates
coverage across weeks, and an archive the budget cuts off goes to the back
instead of starving the tail.
The package arms the timer on every node, once. The sweep runs on the raft
leader and a follower’s fire skips before it reads anything, so a leader
move never leaves the estate unswept. To keep one install out, set
SHC_BACKUP_VERIFY=off when the package is installed; to turn it off later,
systemctl disable --now shc-backup-verify.timer, which an upgrade does not
undo.
shc backup schedule ls on the leader ends with when the sweep last ran and
what it found, and shc doctor run --category backup checks it as
backup.verify_sweep. Each node keeps the report of the sweeps it ran, so
the check asks every node for its report and reads the newest one, and every
node gives the same verdict whichever one leads. It fails when that sweep
disproved an archive, left one unproven for more than 15 days, or picked
archives and proved none of them with no budget cut to blame, and warns when
the estate has held sealed backups for more than eight days and no node holds
a sweep or the newest one is older than that.
Terminal window
$ shc backup schedule ls
...
verify sweep: enabled on shc01, last ran 2026-10-10T12:14:03Z: 12 deployments: 11 proven, 0 disproven, ...
Each run prints a per-deployment verdict table (PROVEN / DISPROVEN /
UNPROVEN when the budget ran out on it / UNVERIFIABLE for plaintext-only
deployments and for a read the daemon could not finish), keeps the latest copy at
/var/lib/shc/backup-verify/last-report.txt, and exits non-zero on any
DISPROVEN row, which parks the unit in systemctl --failed until a later
sweep passes. It also exits non-zero when it picked archives and proved none
of them with no budget cut to blame (every pick ORPHANED or
UNVERIFIABLE), and before it reads anything when a row of
shc ls --all-tenants -o json has an empty tenant, stack or environment. Fires Saturday 12:00 UTC, the quiet half of the clock, hours
from both estates’ 00:45–06:50 capture window.
The budget is the only bound. There is no per-archive size cap: every pick
gets the deep read-back whatever it weighs, because a deep verify streams and
costs the daemon O(1) memory in the archive. Measured against a real
repository, deep reads of 8 GB, 4 GB and 2 GB archives held shcd at 43 MB
peak RSS; the archive-scale memory is the restic child, which is flat in
archive size and exits between verifies. What an archive costs is time and
egress, which the wall-clock budget bounds (3600 s by default). A pick the
budget runs out on is UNPROVEN with X409080VFYUNP, naming its size, the
seconds its read ran and its last proof, and doctor fails once one stays
unproven past 15 days. An estate with archives too big for the hour raises
the budget, and the unit’s timeout with it:
On an estate with more than one location, each location label carries its own
off-site repository, and the archives of a deployment land in the repo of the
location its node is in:
Put each location’s repo outside that location, so losing the location never
loses its backups. The order that decides where an archive goes is: a
backup.repo set at a tenant, environment or stack scope; then the repo of
the deployment’s location label; then the global backup.repo. A location
repo uses the same backup.password and the same backup.aws_* credentials as
the global one, so an archive that was written before a move stays restorable
from the repo its backup row names.
backup.repo may be a path, such as /var/backups/shc/repo. On a cluster of
more than one node that is one repository per node, unless every node mounts
the same storage there: a backup is on the disk of the node that took it and
is lost with that node. shc config set backup.repo <path> says so when it is
set, and shc backup ls then shows a NODE column naming the node that holds
each archive (node and node_id in -o json).
The backup row is cluster-wide, so its id works from any node:
shc backup verify <id> typed on another node runs on the node that holds
the archive.
shc backup export <id> fetches the archive from that node.
shc backup restore --id <id> runs on the node that renders the deployment.
When that is not the node that holds the archive, it fetches the archive
from the holder first, before anything stops, and replays the copy. The
copy goes to a spool file on the restoring node and is removed when the
restore is over, so that node needs free disk for the archive.
A holder that does not answer is the one refusal: X409732BKPELS, before
anything stops, naming the node and the path. Before this, another node
opened its own repository at the path and answered restic’s
wrong password or no key found (SHC-1732).
The form is @<connection>/<bucket>[/<path>]. It works at every scope
backup.repo is read at, and as a location label’s --backup-repo. The bucket
must exist: nothing here creates one.
The key never sits in config. It comes through the s3 grant of the tenant
whose deployment is backed up (the baseline tenant for a scope with none), and
that grant’s --boundary must name the bucket. A tenant with no grant is
refused X403910CONGRT. A grant that does not name the bucket, an unscoped one
included, is refused X403925CONBKT. The connection must be of type s3.
Which key restic is handed depends on the connection’s own key:
Where the endpoint can mint a key for one bucket and the connection’s key
is allowed to (Backblaze B2 with writeKeys, listKeys and deleteKeys,
or Exoscale), SHC mints one key per tenant, restricted to that bucket, and
the connection’s key is not used for the backup. shc connection revoke-grant deletes the minted key.
Where it cannot (a B2 key restricted to one bucket, or a store with no
per-bucket keys such as Hetzner Object Storage), restic is handed the
connection’s own key. It stays in the daemon, and only a bucket the grant
names is opened with it. Prefer a key that is itself restricted to the
bucket.
Archive rows, shc backup ls, shc backup relocate and the doctor all read
the repository as its URL, s3:https://<endpoint>/<bucket>[/<path>]. shc connection rm is refused while a backup.repo names the connection.
The off-provider mirror in 6b reads this form too: the daemon resolves the
bucket URL and the key the grant yields, and hands them to rclone itself.
A verified archive in one bucket is still one bucket. backup.repo is a
single global value: one provider, one credential, holding every archive the
estate has. The exposure that matters is “one repo, no second copy”, and
the package closes it with shc backup mirror and a
shc-backup-repo-mirror.timer/.service pair that runs it: a nightly,
append-only, file-level rclone copy of the whole restic repository onto a
SECOND provider under a SECOND credential. The daemon runs the copy. It
reads each repository and its key the way restic is handed them, a
connection-named backup.repo included, and gives them to rclone in its
environment, so no key passes through the CLI or a shell.
Configure the destination once, in the vault. The daemon derives the S3
endpoint, region and rclone provider from each URL (a connection’s own region
wins):
Each location label that names a --backup-repo gets its own copy on the
same second provider, at <backup/copy_repo>/locations/<provider>/<location>
(_ stands in for a location with no provider), with the same order and
the same floor as the estate repo. One location that fails its copy fails the
run, and the others are still copied. bitspur’s second provider is Backblaze B2
(SHC-1069).
That declaration is the whole arming. The package enables
shc-backup-repo-mirror.timer on every node when it installs, and each fire
decides for itself: a node that is not the raft leader says it skipped and exits
0, and so does a leader whose vault holds no backup/copy_repo. A leader move
therefore needs nothing re-armed, and an estate that never declares a second
provider never writes to one. A locked vault, or a daemon that cannot say who
the leader is, fails the run instead of skipping it.
The package arms the timer once per node and records that in
/var/lib/shc/.backup-repo-mirror-armed, so systemctl disable --now shc-backup-repo-mirror.timer is the lasting opt-out: it sticks across
upgrades and across a remove and reinstall. SHC_BACKUP_REPO_MIRROR=off in the
environment of one install keeps that install from arming it.
It copies; it never syncs.rclone copy has no delete side, and neither
the mirror nor its tests permit one (mirror_2977_test.go fails on any
delete-capable verb or --delete* flag reaching the code). That
is the single most valuable property here: an accidental restic forget --prune against the primary cannot propagate into the copy. The
destination therefore grows monotonically; put a noncurrent-version
lifecycle on the bucket to bound the versioned half, and treat reclaiming
pruned packs as a deliberate operator act rather than a schedule.
The copy order is load-bearing: config, keys, data/, index/,
snapshots/. A repository read in the middle of a write must never see an
index referencing a pack it does not have, so packs land before the index
that names them and the index lands before the snapshots that name it:
every intermediate state of the destination is a valid, merely older
repository rather than a torn one. A run that fails part-way exits non-zero
(parking the unit in systemctl --failed) and says so, and the destination is
still openable: a file-level copy of a restic repo is a restic repo, so
restic -r <copy> snapshots reads it with the same password.
shc backup mirror runs one mirror by hand and lists each repository with
its status and the two snapshot counts its floor compared.
Fires 09:00 UTC, after the 00:45–06:50 capture window and far from the 02:00
retention prune, which takes an exclusive repository lock.
Why rclone and not restic copy. restic takes one set of S3
credentials per process (RESTIC_FROM_PASSWORD exists,
RESTIC_FROM_AWS_ACCESS_KEY_ID does not), and the entire point of the
destination is that it is a different provider under a different
credential. rclone holds per-remote credentials, so it is the one tool in
the payload that can span the two. It is a hard package dependency for the
same reason jq is: the estates install with APT::Install-Recommends "false", so a recommend evaporates on exactly the posture SHC ships to.
This was a hand-dropped script, and that is the defect the packaging
closes. Until 2026-08-20 the mirror existed only as
/usr/local/sbin/shc-backup-repo-mirror.sh on one node of one estate. The
estate rebuilt beside it carried the credentials and none of the machinery
(no script, no unit, no rclone), so it ran on a single bucket, and nothing
reported it: the timer that would have failed was the missing payload. A
node that is rebuilt now gets the mirror from the package.
There IS an in-product way to schedule an encrypted, off-box backup:
shc backup schedule set. Use it. You do not need a host cron.
CORRECTION. Until 2026-08-07 this section read “There is no in-product
way to schedule an encrypted, off-box backup today” and described the
schedule’s fires as “plaintext, on local disk, backup.repo ignored”.
That was false and it contradicted section 3 of this same guide. ce
de396a7 gave the orchestrated path a seal, and shc 27aff725e corrected
section 3 to match, but missed this section, so the guide argued with
itself for three days. The stale text sent operators to build a host
systemd timer they do not need, around a mechanism that already works.
shc backup schedule set registers a schedule whose fires take the
orchestrated path, which seals into backup.repo, encrypted, whenever
a repository and password are configured. It writes a plaintext .bak and
then FINALIZE seals it into the repository, verifies it readable there, and
deletes the local copy. Section 3 has the mechanism; a fire that cannot
seal is skipped or failed, never kept as local plaintext.
shc schedule add schedules an app-declared job from app.yaml, not an
arbitrary command, so it cannot invoke shc backup create. That is
unchanged, and it no longer matters: the backup schedule is the right
surface, not a workaround for it.
What a schedule gets you that section 5’s manual sequence does not: it
runs on its own. Since 0.1.28 a manual create and a fire run the same capture,
with the same quiesce bracket that escalates to a stop past
timeouts.quiesce_hold.
The two things to check before you rely on it, both of which section 3
establishes and neither of which a cron would fix:
backup.repoandbackup.password must both be set, or every fire is
skipped with X400079BKPSKD naming what is missing. A schedule created with
a repo but no password is refused outright (X400077BKPSCD).
Confirm the first fire sealed: shc backup ls --all-tenants and check
encryption=restic / location=restic:… on the new row, exactly as in
section 6. A seal that failed reads failed with X500474BKPSRQ.
Scopes are global, tenant, environment, stack, deployment. A schedule
fires on the cluster leader.
★ DO NOT SCHEDULE AT 02:00, AND THIS IS NOT A STYLE PREFERENCE.retention_cleanup_cron is hard-coded to 0 2 * * *
(ce modules/retention/crons.go, SpecRetentionCleanup) and is leader-only
which is the same node your schedule fires on. Its sweep issues restic forget per
snapshot and then one restic prune per repository, and prune repacks
the repository under a repo-wide EXCLUSIVE lock
(ce modules/backup/archive_delete.go). A capture that fires into that lock
is in direct contention with the pruner for the whole repack.
The examples above use 30 3 for that reason. Keep the 02:00–03:00 hour
clear of every schedule you register, and if you later see a bare
0 2 * * * in a diff, this paragraph is why it should not go back.
The expression grammar, exactly. The parser is
cron.NewParser(Minute|Hour|Dom|Month|Dow|Descriptor)
(ce modules/backup/scheduler.go, cronParser), which is the same parser
the write path’s pre-flight and the live registration both use:
5 fields, minute-first. There is no seconds field: a 6-field
expression is rejected, not silently reinterpreted.
@ descriptors, the whole set: @yearly / @annually, @monthly,
@weekly, @daily / @midnight, @hourly, and @every <duration> (a Go
duration, e.g. @every 6h). Releases up to and including v0.0.27 named only
@hourly / @daily / @weekly in --help; that was an incomplete summary,
never a restriction the parser enforced.
An optional leading TZ=<zone> / CRON_TZ=<zone> prefix; without one the
expression is interpreted in UTC via CE’s core/registry.ScheduleLocation().
No prose."every 5 minutes" is not a schedule; it is a typo that gets
refused.
Anything the parser rejects comes back as X400972INVCRN at
schedule set: the schedule is not written, so a bad expression cannot become
a dead row that never fires and never says so.
Three verb facts that are easy to get wrong:
There is no shc backup schedule get. The verbs are set, unset and
ls (list is a cobra alias for ls). A shc.backup.schedule.get route
does exist on the daemon, but it is an API surface (pkg/apiclient’s
BackupScheduleGet, for the terraform provider) with no CLI spelling.
shc backup ls refuses positional arguments (cobra.NoArgs). Filter
with --stack / --status / --all-tenants; shc backup ls <stack> is an
error, not a filter.
unset converges, it does not stop the schedule immediately. The handler
unregisters the live cron on the node that served the request and deletes
the row. If a follower served it, the leader’s in-memory cron keeps firing
until its own reload lands; the deleted row is what makes it converge on the
next reload, leader transition, or restart. shc backup schedule ls reads the
backup_schedule table, so it goes quiet immediately, before the leader
does. If you need the fires to stop now, unset against the leader.
Two things to know before you pick the expression. A fire holding its freeze
past timeouts.quiesce_hold (300 seconds) will stop the stack, retake the
capture against the halted containers, and start it again: deliberate, because
an archive nobody can trust is worse than a short outage; the stop is reported
on the row and in StackStopSeconds. And section 4a applies in full: a
scheduled capture of a database is a window in which dependent stacks see
errors, not latency, so schedule it where you would accept that.
Until a fire is confirmed sealed, keep treating {state_home}/backups as
secret material: an unsealed .bak is readable without a key (section 3).
Retention is ARMED, and it has a shipped default. Something prunes on its
own: set the policy you actually want BEFORE your first capture, not after.
CORRECTION. This section previously said “nothing prunes on its own.”
That was wrong. Verified in ce: modules/retention/crons.go registers
retention_cleanup_cron on the spec 0 2 * * * as a leader-only cron,
and the queue-side handler it enqueues is bound in
daemon/services_seams_retention.go. It is wired end to end.
That 02:00 slot is spoken for, and it is why section 7 tells you to schedule
elsewhere. The sweep’s restic prune takes a repo-wide exclusive lock, on the
same node your backup schedule fires on. Leave the 02:00–03:00 hour to the
pruner.
DefaultRetentionPolicy() (ce modules/backup/types.go, mirrored by
retention.DefaultPolicy() in modules/retention/types.go) is what an
unconfigured tenant gets:
keep
daily
7
weekly
4
monthly
12
yearly
5
So the cron begins acting at the first 02:00 after archives exist, against
those numbers, whether or not you ever ran shc retention set. The bucket does
not grow without bound, but neither does it keep what you assumed it would.
Restic deduplicates, so a daily capture of the 5 GB openobserve stack costs
far less than 5 GB per day; that is a reason to consider keeping more than
the default, not a reason to leave it unset.
Set it explicitly before the first capture. An explicit policy is the
difference between a retention window you chose and one you inherited, and the
first 02:00 comes sooner than a review does.
shcretentionset--daily7--weekly4--monthly12--yearly5# the shipped default, stated
shcretentionpreview# what would be removed
shcretentionapply# run it now rather than waiting for 02:00
shc retention ls shows the resolved counts, so run it once after setting to
confirm the scope you set is the scope that resolves.
Two cautions carried over from how this repository handles keys and archives:
Rotating the restic password orphans every archive taken under the old one.
There is no key-add verb. Decide the key once.
shc backup verify is a membership probe, not an integrity check. Restic’s own
check is the whole-repository verification, and it takes an exclusive lock:
run it outside a capture window.
Restic encrypts client-side under backup.password, and the two hel1 estates
hold different passwords, so neither can read the other’s archives. That is
not the same as isolation. Confidentiality is not availability: the S3
credential decides who can delete, and a delete needs no key at all.
Both hel1 estates were installed with the SAME Exoscale credential, and it was
an Owner-role key, Exoscale’s unrestricted role, whose policy is literally
{"default-service-strategy":"allow","services":{}}. Measured on 2026-08-04,
that one key sitting in each estate’s vault could list, write and delete objects
in the other estate’s backup bucket, and in bitspur-k8s-archive, a bucket
that has nothing to do with either estate and holds a content-verified k8s dump
set. Anything that could read either vault inherited all of it.
One S3 credential per estate, scoped to that estate’s own bucket. Not one
per organisation, and never the org-Owner key. A backup credential’s whole job
is s3:<endpoint>/<one-bucket>; anything wider is blast radius with no
corresponding capability.
On Exoscale that is an IAM role whose policy denies by default and allows
SOS only for one bucket, plus an API key bound to it. The expression variable is
parameters.bucket; resources.sos.bucket and resources.bucket are both
accepted by the API and then match nothing, which denies the credential its
own bucket. Verify, do not assume:
There is no shc backup key set. shc backup key offers only ls / rm /
rotate, all of which act on the restic repository password, not on the
object-store credential, and rotate refuses when the current password is
empty, so it is not an install path either. Do not reach for it: rotating the
restic password orphans every archive taken under the old one and there is no
restic key add verb (section 8).
The object-store credential has no special verb because it does not need one. It
is an ordinary secret plus an ordinary config key:
Terminal window
# 1. the secret, into the vault at global scope (the vault must be unlocked)
shcvaultsetbackup/aws_access_key_id'<EXO…>'
shcvaultsetbackup/aws_secret_access_key'<secret>'
# 2. the config keys, as references, never as literals
Note the two spellings of the same thing: the vault key is
backup/aws_access_key_id (what shc vault ls --scope global shows), and
the config reference is ref+vault://shc/system/backup/aws_access_key_id
(shc/system/ is the scope prefix in the ref grammar). shc vault get shc/system/backup/aws_access_key_id answers X404907SECNOF, which reads like a
missing secret and is only a mis-spelled path.
Both keys resolve on demand through the secret dispatcher (LoadEnv in
daemon/services_backup_config.go maps them onto AWS_ACCESS_KEY_ID /
AWS_SECRET_ACCESS_KEY for restic), so replacing a credential is a shc vault set and nothing else. No redeploy, no daemon restart. A locked vault makes
every backup fail rather than fall back, which is intended, and means the
credential swap wants the vault unlocked (shc vault -p).
Do not accept “the policy looks right”. Two probes settle it, and neither can
destroy anything even if the policy is wrong:
DeleteObject on a key that does not exist in the other estate’s bucket.
S3 DeleteObject is idempotent: an authorized delete of a missing key returns
204 and destroys nothing, an unauthorized one returns 403. Decisive, zero
blast radius.
DeleteBucket without -r against a bucket that is not empty. An
authorized call can only answer BucketNotEmpty; it can never delete.
Run every probe against a credential you know is unrestricted as well. A
DENIED from a probe that never issued a request proves nothing: the first
attempt at this measured exo storage delete without -f and recorded seven
“allowed” verdicts that were actually confirmation prompts hitting EOF.
Measured with the scoped keys in place:
operation
own bucket
other estate’s bucket
bitspur-k8s-archive
ListObjects
allowed
403
403
PutObject
allowed
403
403
DeleteObject
allowed
403
403
DeleteBucket
n/a
403
403
and the same matrix under the Owner key it replaces: allowed, allowed, allowed
everywhere, with only BucketNotEmpty standing between it and a bucket delete.
A credential scoped to one bucket still has full delete rights inside that
bucket. It has to, because restic forget/prune rewrite the repository. So
per-estate scoping bounds the damage to one estate’s own archives; it does not
make them immutable. If archives need to survive their own estate being
compromised, that is object-lock / versioning on the bucket, which is a separate
piece of work and is not wired.
10. Whose data is in this archive? The boundary audit
The ruling. A stack’s backup captures the data that stack owns. It never
follows an integration edge to capture data owned by the thing on the other
side. Nextcloud must never back up its S3 bucket: that bucket is reached
through an external-s3 edge and is not Nextcloud’s data. The provider owns its
own durability story; crossing the edge duplicates another stack’s data under
the wrong name and produces archives whose size is unrelated to the capturing
stack’s real state.
What it cost when it was crossed.corp/hz-s3 is an external-s3
connector: services: {}, no volume, no database, no bucket of its own. With
backup.buckets armed it pulled the Nextcloud bucket cloud-bitspur-com
through a container into restic on 2026-08-13/14/15. Those captures blew the
30-minute converter budget, exceeded the ar(5) 10^10 size ceiling and were
silently truncated to about 10%. An empty backup.buckets is the CORRECT state.
It names each package OWNS-AND-CAPTURES / CROSSES-A-BOUNDARY /
CAPTURES-NOTHING / AMBIGUOUS, and for a crossing prints the evidence, the
variables that arm it and what removing it would cost.
Run the estate half, not just the catalogue. Whether a crossing is armed
is decided by vars this deployment was installed or updated with, and the
shipped package says nothing about it; the catalogue file was in its correct
state throughout the hz-s3 incident. The estate half resolves each converter’s
capture-set expression against the deployment’s own app/vars.yaml, and
separately compares each stack’s archives against what that stack itself
stores.
Two things it is not. It is not a CI gate and nothing fails because of it:
it is something you run when you want to know. And it never disarms anything:
every command it issues is a read, because turning off a running backup is a
decision about what depends on it, not one a report gets to make.
backup.buckets being empty does not prove an external-s3 stack is
disarmed. The converter falls back to the bucket serve pin, so
--var bucket=<name>, set for access-control reasons, with the word backup
nowhere in the command, arms a full-corpus capture by a second route. The
report checks both.
11. What the estate says when none of this is armed
Every section above describes something an operator has to do. Each of the
five defects measured on 2026-08-20 was a section somebody had not done, and
in every case the product held the information needed to notice and never
looked. Four shc doctor lines close that, all under
shc doctor run --category backup, and all of them are also emitted once a
day by the backup_coverage_audit cron (08:00, leader-only) so they reach
whatever notification rules the estate has rather than waiting to be pulled.
Check
Asks
Fires when
backup.encryption
is the archive encrypted with a key only you hold
no key, half a key, or a key that will not resolve
backup.destination
can a fire actually reach the repository
a cloud repo whose credentials are declared and do not resolve (error), or which needs credentials SHC has none configured for (warning)
backup.armed
is what the estate declared actually armed
backup/copy_* in the vault with no mirror unit armed on the leader; schedules registered over an estate that has never captured anything
backup.coverage
does each deployment have an archive
never protected, last attempt failed, stale, or object data in a bucket no capture reads
backup.orphans
do the ledger and the bytes still refer to each other
a row that records no location; an archive file under the backups root belonging to no row
Two of these are worth reading together, because on 2026-08-20 they said
opposite things about the same estate. backup.encryption reported healthy
on promanager-new: backup.repo was an s3: URL and backup.password
resolved out of the vault, which is exactly the question that check asks.
The two backup.aws_* rows were ref+vault:// references to keys the vault
had never held, so every scheduled fire was skipped before it opened a
row: no archive, no failed row, no warning, five schedules, days of silence.
backup.destination is the line that says so, and backup.encryption’s
healthy line now points at it rather than reading as an all-clear.
backup.armed is the second-copy half of section 6b. An estate that has
stored backup/copy_repo has declared it wants two providers; if nothing
is armed to deliver that, the declaration reads as protection and is not.
The check is silent on estates that never asked for a second copy, and says
unknown rather than no when the vault is locked.
None of the four repairs anything. shc doctor run --fix will not arm a
timer that starts paying a second object store, will not write a credential,
and will not delete an archive; each hint names the capability and the
operator performs it.
The replicas of a service bind the same mounts, so a backup of one has
no copy to choose. backup.yaml takes no where: or prefer:: a file
that sets either is refused (X400258BKPSEL) before anything is
captured, and the refusal names the file and the key. Distinct copies of
a service are a group of members.
A database that runs as a group of members keeps one dataset in every
member, so the converter that captures it says from: <selector>, for
example from: group=postgres, and each backup reads one member: the
lowest-numbered member it matches that is ready and is not the writer,
else the writer. The writer is the member the group-named alias with
:1 picks, postgres’s role=primary, so
from: group=postgres & role!=primary never reads the writer. A member whose pre_backup refuses, such as a standby that is
behind, is passed over for the next one. The archive holds that one copy, never one per member or per
location. With no member to read the backup is refused (X409895BKPGRP)
and the refusal says why each member was passed over. An app’s from:
uses only group, member, service and the labels the app writes
itself, which hold on every estate (X400259BKPFRM otherwise). A restore writes the copy into the
lowest-numbered member and empties the others, which clone from it. A
group whose members hold distinct data names no from:, and every
member is captured. See Database apps, HA by scale.
A replica keeps its mounts on the node that runs it. A backup captures
them there, whichever node the backup itself runs on: the node a schedule
fires on, the node a clone or move takes its backup on, or the node shc backup create reached. That node pauses the replica’s containers for its
own capture and says whether the pause held, and the archive’s
point_in_time and not_quiesced marker come from that answer. The node
running the backup pauses nothing when no replica of the deployment runs
on it.
A backup never completes without a replica’s data. It refuses, or ends
failed, with X409899BKPRPL naming the replica, its node and why:
the replica’s node does not answer, or answers that it holds no such
replica
the capture on that node fails
a service that is not stateful runs on several nodes and each holds its
own copy of the same mount, and an archive carries one copy
A mount backup.yaml doesn’t list under include, whether it’s under keep
or scratch, is not data the backup owes, so it never causes this refusal.
shc backup create --file reads only the disk of the node it is written on and
is refused when a replica’s data is on another.