Skip to content
SHC Docs

Backup schedules: arming the subsystem that is built, idle, and one config row away

Status: open design. A schedule is a row in the backup_schedule table, and that part is built. The cadence (§3), the retention pin (§4) and the runbook (§6) are not signed off.

As written: MIXED. The cadence proposal (§3), the retention pin (§4) and the runbook (§6) are still PROPOSED and still want owner sign-off. But §1 and §5 describe a CONFIG-BACKED storage model that has since been SUPERSEDED; a backup schedule is now a row in the backup_schedule TABLE, and the backup.schedule config path is RETIRED. Do not draft a step from §1’s terraform-reachability table; every shc_config-writes-a-schedule statement in it is now refused by the daemon.

The title’s “one config row away” is itself the stale phrasing: it is one shc backup schedule set away, and there is no config row involved.

Statuses re-verified 2026-08-07 against ce df1e7e4, the pin shc v0.0.27 carries. The prose below is left as authored; this header and the per-section notes are the authority where they disagree.

§What it saysStatus
The problem, measuredNeither estate has a schedule; the subsystem is built and idleSTILL TRUE (as measured 2026-08-04)
§1A schedule is “an ordinary scoped config row” at backup.schedule, loaded by ListPathAcrossScopes, writable through shc_config at global/tenant scopeSUPERSEDED. Schedules moved to the backup_schedule table under the 2026-08-07 config-vs-state ruling. ScheduleConfigPath is now a RegisterRetiredPath: writes are refused with X409073CFGRTD at Service.Set and at the daemonless overlay. Nothing reads the path any more: the loader’s carry of leftover rows was removed with SHC-3034, so the refusal is all that remains of it.
§1’s guard-bypass argument”Writing the row directly through shc_config bypasses both guards”RESOLVED, not just argued. The bypass is closed by the retirement; that is precisely why the storage moved. The argument is correct history; it is no longer a live hazard.
§1’s VerdictRetention in terraform now; schedules via CLI then a provider resourceSTANDS, on the new reasoning
§2 SequencingCredential posture firstSTILL PROPOSED; blocked on estate access
§3 the schedule set, and the 02:00 collisionThe cron cadence proposalSTILL PROPOSED. The 02:00 collision it identifies is CONFIRMED and has been carried into the shipped guide.
§4 RetentionPin the shipped defaultsSTILL PROPOSED
§5 shc_backup_schedule resourceThe provider resource, and the daemon facts it rests onSTILL NOT BUILT. Its apiclient table is correct: BackupScheduleSet / Get / Unset / ListGet all exist and .get has no CLI spelling. Its unset-converges caveat is correct and verified. But its framing of the row as config, and its “extending shc_config with stack/environment” alternative, are SUPERSEDED: there is no config row left to extend toward. Its cron-grammar sentence is incomplete; see the note at §5.
§6 RunbookWhat an operator runsSTILL PROPOSED; shc backup schedule list in it is a valid alias of the canonical ls
§7 / Open questionsWhat lands, what waitsSTILL OPEN

See Config versus state: the migration, whose first migration is this one, and the shipped Enable backups §7 for the operator-facing grammar and the 02:00 rule.

Neither estate has a single backup schedule in any tenant. Measured read-only on both leaders on 2026-08-04 and recorded in docs/guide/operations/enable-backups.md §1: shc backup schedule list returns no backup schedules on promanager and on bitspur. shc backup ls returns five rows on promanager (every one a pre-uninstall safety archive, every one ENCRYPTION none, every one naming a stack that no longer exists) and no backups on bitspur.

Production data on both estates has no automated backup at all, while the backup subsystem sits fully built: two capture engines, a seal into object storage, a quiesce bracket with escalation, a retention pruner on a daily cron, and a leader-only scheduler subsystem (backup_scheduler, priority 70) that loads its schedule set on become-leader and has never had one to load.

§2 of that guide already established the cause and it has not changed: configuration is complete, the vault resolves, the buckets exist and are reachable. No schedule was ever registered. That is the whole of it.

1. The deciding question: terraform-able, or CLI-only?

Section titled “1. The deciding question: terraform-able, or CLI-only?”

★ SUPERSEDED: this section’s premise was reversed within the day. A backup schedule IS a table row now: backup_schedule, per the 2026-08-07 config-vs-state ruling (ce modules/backup/schedule_store.go). The backup.schedule config path is RETIRED: shc config set backup.schedule=…, the -c install/update passthrough, the daemonless overlay and the shc_config terraform resource are all refused (X409073CFGRTD). The reachability table below is therefore a table of things that no longer work; only the “stack / environment / deployment scope: No” rows survive, and they survive for a different reason (there is no config row at any scope). Reads and deletes on the retired path still succeed, and the loader still honours a leftover legacy row so an operator can see it, let it fire, and clean it up. The section’s conclusion is unchanged; its mechanism is not.

Both, and the split matters. A backup schedule is not a table row and not an API-only object: it is an ordinary scoped config row, which is why part of this is reachable from the pipeline today and part is not.

modules/backup/scheduler.go:

const ScheduleConfigPath = "backup.schedule"

The schedule’s target is not encoded in the key. It is the config row’s own scope columns. subsystems/impl/backup_scheduler.go loads the set with

rows, err := svc.ListPathAcrossScopes(ctx, backup.ScheduleConfigPath)

and ListPathAcrossScopes is WHERE path = ?, an exact path match, not a prefix walk. So every schedule in the estate is a row at exactly backup.schedule, distinguished from its siblings only by (tenant_name, stack_name, environment). BackupScopeFromConfigEntry accepts two value shapes: a bare cron string, or a map {cron, all_connected, with_dependants}. shc backup schedule set writes the map form (modules/backup/router.go).

Reachable from terraform todayNotes
Retention policyYes, fullybackup.retention.{daily,weekly,monthly,yearly} is ordinary scoped config; shc_config sets it at global or tenant scope
Backup schedule, global scopeYes, as a bare cron stringshc_config with key = "backup.schedule"
Backup schedule, tenant scopeYes, as a bare cron stringshc_config with tenant = "corp"
Backup schedule, stack / environment / deployment scopeNoshc_config exposes key, value, yaml, tenant and nothing else; there is no stack or environment attribute (terraform/provider/internal/provider/config_resource.go)
all_connected / with_dependantsNothe yaml form requires every leaf to be a scalar, and would flatten to backup.schedule.cron, a different path, which the exact-match loader never sees

There is a sharper objection than the missing scopes. Writing the row directly through shc_config bypasses both guards the .set endpoint applies. scheduleSetHandler calls Scheduler.RegisterScope first (which validates the cron expression and returns X400972INVCRN on a bad one) and Service.CheckScheduleDestination, which returns

X400077BKPSCD refusing to create a backup schedule: backup.repo (…) is configured
but backup.password is not, so every scheduled archive would be
written UNENCRYPTED to local disk instead of the repository

A raw config write gets neither. A typo’d expression lands as a dead row that never fires and never says so, and the one refusal that exists specifically to stop a cron from writing plaintext forever is not consulted. Using shc_config to arm a backup schedule trades away the safety mechanism this whole subsystem’s credibility rests on.

  • Retention: land it in terraform now. It is ordinary config, the scopes line up, and there is no guard to bypass.
  • Schedules: CLI for the first arming, then a provider resource. The runbook in §6 is what an operator runs. The durable answer is a shc_backup_schedule resource (§5), which is a pure provider-side addition with no daemon change, because pkg/apiclient already carries BackupScheduleSet, BackupScheduleUnset, BackupScheduleGet and BackupScheduleListGet. That is a complete CRUD surface; the provider simply does not use it yet.

Note also that neither ops repo sets any backup.* key in terraform. Both estates’ backup.repo, backup.password and backup.aws_* values were set by hand, out of band. Read from origin/main via the GitLab API: promanager/ops and bitspur/ops each carry one shc_config resource, and it wires dns.public_address only. The estate’s entire backup posture is currently outside the pipeline that is supposed to own it.

2. Sequencing: the credential posture comes first

Section titled “2. Sequencing: the credential posture comes first”

Arming a schedule before §2 is complete automates the production of compromised archives. Four separate facts force this ordering.

a. The shared S3 credential. Both hel1 estates were installed with the same Exoscale key, and it was an Owner-role key, policy literally {"default-service-strategy":"allow","services":{}}. Measured 2026-08-04: either estate could list, write and delete objects in the other’s backup bucket, and in an unrelated bitspur-k8s-archive bucket. Restic’s client-side encryption does not help here: confidentiality is not availability, and a delete needs no key. Backup isolation between the two estates does not exist. Guide §9 carries the scoped-role recipe and the non-destructive probe matrix that proves it took.

b. The restic repos must be DELETED, not rotated. restic key passwd re-wraps the same master key under a new password. A repository whose old password was public stays readable by anyone who ever held it, forever, after any number of rotations. There is no restic key add verb in this product and no key-rotation story that preserves archives. The only correct remediation for a repo created under a published key is to destroy it and create a new one.

Two published keys are in scope. The retired constant "shc-default-backup-password" shipped in a public AGPL repo and was the daemon’s invented default until it was removed on ce origin/main; docs/reference/legacy-and-breaking.md states the irreversible consequence: archives taken under it must be treated as unencrypted. The second is subtler: Load used to return the raw config row, so an operator who followed the shipped guidance and set backup.password: ref+vault://backup/password encrypted under the literal 27-byte string ref+vault://backup/password, also a constant published in this repo, and indistinguishable from a real key at every surface. Both estates carry backup.password = ref+vault://shc/system/backup/password, so both are in the blast radius of the second one for any repository initialised before that fix.

The measured good news: the only repository either estate is known to hold on a published key is the local one at /var/lib/shc/backups/, created 2026-07-27, with zero snapshots (guide §2). Deleting it costs nothing. Whether a repository exists in either S3 bucket at all is unestablished: no capture has ever sealed into one, and restic init runs at first capture. If the buckets are empty, this remediation is nearly free, which is the one advantage of having never taken a backup.

c. Existing .bak archives carry live credentials in cleartext. An unsealed orchestrated .bak is a plain !<arch> container: manifest.json plus an uncompressed tar, no key needed for either. It carries the deployment’s install-time --var values verbatim and the app’s own credential files: a Nextcloud config.php dbpassword, a frappe site_config.json encryption_key, a postgres converter’s pg_dumpall role lines. Separately, InstallVdeployment resolved ref+vault://…?autogen in place and persisted the resolved value into the instance’s manifest.json, twice. ce 65789fa fixes the mechanism (vvars stays ref-intact and the vplug data block is rendered twice, once resolved for the jobs and once ref-intact for the sidecar), but it does not remediate a sidecar already written, and it does not remediate an archive already taken.

This is a hard prerequisite for bitspur specifically. bitspur/ops declares three shc_vdeployment resources; promanager/ops declares none. Scheduling bitspur on a daemon that predates 65789fa puts cleartext vault credentials into every archive, on a cron, unattended. shc origin/main already pins a ce carrying the fix (c50f7e8, 2026-08-07 09:14, which contains 65789fa at 07:11), so the prerequisite is a roll, not a code change. The last recorded estate version is v0.0.22, which predates it; re-derive that before acting on it.

d. Nothing prunes today, and the .bkg bundles never will. Retention has never run against a real archive on either estate. {state_home}/exports/*.bkg group bundles are written by decrypting members back out and are never pruned by retention; the guide says delete them yourself.

  1. Roll both estates to a build at or past shc origin/main’s current ce pin. Mandatory for bitspur (three vdeployments); good hygiene for promanager.
  2. Mint one Exoscale IAM role + API key per estate, scoped to that estate’s own bucket. Install via shc vault set + shc config set (guide §9). Prove the boundary with the DeleteObject / DeleteBucket probes; a DENIED from a probe that never issued a request proves nothing.
  3. Establish what is actually in each bucket. If any repository predates the password-ref fix, empty the bucket and let the next capture restic init a fresh repository. Do not restic key passwd.
  4. Delete /var/lib/shc/backups/ on both leaders: zero snapshots, published key, nothing writes to it.
  5. Prove one manual encrypted capture end to end (guide §5.2 → §6): the row must read encryption=restic and location=restic:…, and shc backup verify must pass.
  6. Only then arm the schedules.

3. The proposed schedule set: needs sign-off

Section titled “3. The proposed schedule set: needs sign-off”

Two constraints fall straight out of the code and both rule out the obvious answer.

The documented example collides with the retention prune. modules/retention/crons.go sets SpecRetentionCleanup = "0 2 * * *"; the retention cleanup cron is leader-only, daily, at 02:00. Its sweep issues restic forget per snapshot plus one restic prune per repository, and prune takes an exclusive repository lock. The cron expression used as the example in shc backup schedule set --help, in guide §7, and in this project’s own prose is "0 2 * * *". Following the documented example puts every scheduled capture in direct lock contention with the pruner. Nothing in the proposed set fires between 02:00 and 03:00.

A quiesce is a hard error to dependent applications, not latency. Guide §4a is explicit: ALTER SYSTEM SET default_transaction_read_only = on does not queue writes, it fails them, and super_read_only and Nextcloud maintenance mode behave the same in their own vocabulary. A scheduled capture of a shared database is, for its duration, an outage the dependent stacks see as 5xx. Cadence and window are therefore a business decision, not a technical one, which is why this section needs a signature.

A third consideration is specific to tenant scope. timeouts.quiesce_hold defaults to 300 seconds; a fire that holds its freeze past the deadline stops the stack, retakes the capture against halted containers, and starts it again. Unattended, at night. Two stacks are candidates: openobserve (~5 GB) and, on any tenant-scope sweep of master, traefik, which owns :80/:443, so pausing it is a total estate ingress outage for the length of the capture.

ResolveScopeStacks expands a tenant-scope fire to every stack under the tenant, and every stack in that fire shares one consistencyID, so shc recover --consistency-id <id> restores them as a group, DB-provider-first. Capture order within the sweep is the deployment table’s order and is not DB-first; the consistency group, not the capture order, is what makes the restore correct.

That gives tenant scope a property no per-stack set has: a deployment added next month is protected the day it is created, with no terraform change and nobody remembering. Protection by omission is precisely how both estates arrived at zero archives. The proposal weights that heavily.

ScopeTargetCronRationale
tenantcorp30 3 * * *The application tenant, where deployments get added. Tenant scope so new stacks are covered automatically. One consistency group per fire.
stackmaster / openobserve0 4 * * *~5 GB, the long one. Its own window so it cannot push a shared sweep past the hold deadline.
stackmaster / postgres30 4 * * *Carries the read-only quiesce and backs Keycloak’s realms. Its own window, after the app tenant.
stackmaster / minio0 5 * * *Blob store for huly; pre_backup exports its IAM database into the data volume.
stackmaster / traefik30 5 * * *Ingress outage for the duration. Deliberately last, deliberately its own row, deliberately the quietest minute.

master gets explicit per-stack rows rather than a tenant-scope row for one reason: a tenant sweep of master would pause traefik inside a sweep whose duration is the sum of every other stack in the tenant, and there is no exclusion mechanism in the scope model. master holds system stacks and changes rarely, so the “silently unprotected” risk that argues for tenant scope in corp is much weaker there.

Open for the owner:

  • Window. 03:30–05:30 UTC is a guess, not a measurement. What hours are these estates genuinely quiet, and is a short 5xx window in them acceptable?
  • Cadence. Daily is proposed. Weekly halves the outage count and quadruples the worst-case loss.
  • traefik at all? Its ~270 MB is the ACME store. Losing it means re-issuing certificates, recoverable without a backup. Trading a nightly ingress blip for that may not be worth it.
  • --all-connected? Not proposed. On a tenant-scope fire it is redundant, and on the master rows it would pull peers into windows chosen for one stack.

4. Retention: pin the defaults, do not change them

Section titled “4. Retention: pin the defaults, do not change them”

The shipped defaults are daily 7, weekly 4, monthly 12, yearly 5 (modules/retention/types.go), resolved per field through a Stack > Tenant > Global cascade. They are already in force: retention needs no configuration to run, and the cleanup cron has been firing daily at 02:00 against an empty archive set this whole time.

The proposal is to write those same numbers into terraform rather than change them. That makes the policy reviewable in git, immune to a default change in a future release, and (unlike the schedule rows) it is ordinary config with no guard to bypass, so shc_config is the correct instrument.

hetzner/hel1/deployments/backup.tf
resource "shc_config" "backup_retention" {
yaml = <<-EOF
backup:
retention:
daily: 7
weekly: 4
monthly: 12
yearly: 5
EOF
}

At one capture per stack per day this keeps up to 28 archives per stack, which restic deduplicates aggressively. Do not raise these before step 4 of §2 has proven a real capture; retention deletes archives, and a pruner pointed at a repository nobody has verified is a way to lose the only copy.

One correction belongs with this: core/config/default.yaml describes backup.retention as a “Restic forget policy” whose nulls make “restic fall back to its built-in defaults.” Both halves are false. No restic forget policy exists anywhere in the tree: there is no --keep-daily, --keep-weekly, --keep-last or --keep-monthly on any call; the only forget call site is a bare per-snapshot restic forget <id>. And a null does not defer to restic, it selects SHC’s own 7/4/12/5. An operator choosing a retention policy from that comment is choosing from a description of a mechanism that does not exist. A one-hunk correction lands with this proposal.

5. shc_backup_schedule: the durable answer

Section titled “5. shc_backup_schedule: the durable answer”

Still NOT BUILT, and two of its statements need correcting.

  • The storage moved. This section’s shc_config alternative is moot: backup.schedule is a retired config path, so there is nothing to extend shc_config toward. Going through .set is no longer merely “the entire point”; it is the only way in.
  • The cron-grammar sentence is incomplete. cron.NewParser(Minute|Hour| Dom|Month|Dow|Descriptor) accepts the full descriptor set, not three: @yearly/@annually, @monthly, @weekly, @daily/@midnight, @hourly, and @every <duration>, plus an optional leading TZ=/CRON_TZ= prefix. What the sentence gets right is the rest: 5 fields, no seconds field, no prose. A rejected expression is X400972INVCRN.
  • The unset-converges caveat is CORRECT and worth keeping, verified at ce modules/backup/router.go, scheduleUnsetHandler. A follower-served unset deletes the row and unregisters only its own cron; the leader keeps firing until its reload lands.
  • The apiclient table is CORRECT. All four calls exist, and BackupScheduleGet really is API-only; shc backup schedule get is not a CLI verb (the verbs are set / unset / ls, with list as an alias).

Specified here, not built, pending sign-off on §3.

The provider exposes thirteen resources; none is backup-related. The addition is small and needs no daemon change, because the generated client already has the whole surface:

Provider operationapiclient callEndpoint
Create / UpdateBackupScheduleSetPOST /api/method/shc.backup.schedule.set
ReadBackupScheduleGetPOST /api/method/shc.backup.schedule.get
DeleteBackupScheduleUnsetPOST /api/method/shc.backup.schedule.unset
Import / driftBackupScheduleListGetGET /api/method/shc.backup.schedule.list
resource "shc_backup_schedule" "corp" {
scope = "tenant" # global|tenant|environment|stack|deployment
tenant = shc_tenant.corp.name
cron = "30 3 * * *"
}
resource "shc_backup_schedule" "openobserve" {
scope = "stack"
tenant = "master"
environment = "default"
stack = "openobserve"
cron = "0 4 * * *"
}

Going through .set rather than shc_config is the entire point: the resource inherits RegisterScope’s cron validation and the X400077BKPSCD refusal, so a bad expression or a half-configured key fails at plan/apply time with a named error instead of becoming a row that quietly never fires.

Two things to get right in the implementation. unset unregisters the live cron only on the node serving the request; if a follower serves it, the leader keeps firing until its own config reload, and the config-row deletion is what makes it converge; Delete should therefore verify through BackupScheduleListGet rather than trust the 200. And the scheduler’s parser is stricter than shc schedule add’s: cron.NewParser(Minute|Hour|Dom| Month|Dow|Descriptor): 5-field cron and @hourly/@daily/@weekly only, no "every 5 minutes" prose and no seconds field.

Extending shc_config with stack and environment attributes is the tempting cheaper alternative. It should not be taken. It would widen the raw-config write path to exactly the scopes where the guards matter most, and the guard bypass in §1 is the reason not to.

Estate SSH is not reachable from the box this was authored on, so none of this was executed. Run on the leader, with the vault unlocked. Substitute the real deployment list from shc ls --all-tenants; the “12 deployments” figure in guide §1 is stale and bitspur has grown well past it.

Terminal window
# --- preflight -------------------------------------------------------------
shc ls --all-tenants # derive the REAL stack list; do not assume 12
shc backup schedule list # expect: no backup schedules
shc retention ls # expect: 7 / 4 / 12 / 5
shc doctor run -c backup
shc vault status # must be unlocked

Do not proceed past here until §2 steps 0–4 are complete and one manual capture has come back encryption=restic / location=restic:… and passed shc backup verify.

Terminal window
# --- arm (order does not matter; the expressions do) ------------------------
shc backup schedule set "30 3 * * *" --scope tenant -t corp
shc backup schedule set "0 4 * * *" --scope stack -t master -s openobserve
shc backup schedule set "30 4 * * *" --scope stack -t master -s postgres
shc backup schedule set "0 5 * * *" --scope stack -t master -s minio
shc backup schedule set "30 5 * * *" --scope stack -t master -s traefik
shc backup schedule list # five rows, each with a next_run

--scope stack is the default and -t/--tenant is the inherited global flag; both are stated explicitly above because the tenant is resolved client-side (-t → $SHC_TENANT → the persisted shc ctx -t pointer → "default") and a schedule that lands on the wrong tenant looks identical to one that landed on the right one.

Terminal window
# --- verify the FIRST fire, the next morning --------------------------------
shc backup ls --all-tenants -o json | head -40

Every new row must read encryption=restic and a location starting restic:. A row reading none means the seal fell back to local plaintext; check the logs for X200075BKPSEL (the reason) and X200079BKPLOC (the path), and treat {state_home}/backups as secret material until it is fixed.

Terminal window
# --- ground truth, independent of SHC ---------------------------------------
export AWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=…
export RESTIC_REPOSITORY='s3:https://sos-de-fra-1.exo.io/shc-backup-<estate>-hel1'
export RESTIC_PASSWORD=…
restic snapshots # one per stack captured
restic check # exclusive lock: never inside a window

shc backup ls showing completed is not proof the archive exists; guide §6 demonstrates a deleted snapshot still listing as completed. Use shc backup verify <id>, and restore-test one stack before trusting the set.

To back out, shc backup schedule unset --scope <same> -t <same> [-s <same>] with the scope reproduced exactly; the scope tuple is the row’s identity.

7. What lands with this proposal, and what does not

Section titled “7. What lands with this proposal, and what does not”

Lands now: neither touches an estate and neither depends on a decision:

  • this document;
  • the core/config/default.yaml retention-comment correction (§4).

Blocked on owner sign-off:

  • the cadence, window and retention numbers in §3 and §4;
  • the shc_backup_schedule provider resource (§5), whose shape follows from the scope set that gets signed off;
  • the backup.tf root in each ops repo: retention is landable immediately, the schedule rows want the resource.

Blocked on estate access: every step in §2 and §6.

  1. Window and cadence (§3). The single decision everything else waits on.
  2. Is a nightly ingress blip for traefik worth its ACME store? (§3)
  3. What is actually in the two S3 buckets? Determines whether §2 step 2 is free or expensive. Unestablished from here.
  4. timeouts.quiesce_hold = 300s: raise it, or accept that a stack whose capture runs long gets stopped and restarted unattended? openobserve is the live candidate.
  5. Should the backup posture move into the ops repos at all? Today the keys are entirely out of band. Retention and schedules can be codified; the vault secrets deliberately cannot, and must never enter terraform state.