Backup schedules: arming the subsystem that is built, idle, and one config row away
Status: open design. A schedule is a row in the
backup_scheduletable, and that part is built. The cadence (§3), the retention pin (§4) and the runbook (§6) are not signed off.
As written: MIXED. The cadence proposal (§3), the retention pin (§4) and the
runbook (§6) are still PROPOSED and still want owner sign-off. But §1 and §5
describe a CONFIG-BACKED storage model that has since been SUPERSEDED; a
backup schedule is now a row in the backup_schedule TABLE, and the
backup.schedule config path is RETIRED. Do not draft a step from §1’s
terraform-reachability table; every shc_config-writes-a-schedule statement in
it is now refused by the daemon.
The title’s “one config row away” is itself the stale phrasing: it is one
shc backup schedule set away, and there is no config row involved.
Statuses re-verified 2026-08-07 against ce df1e7e4, the pin shc v0.0.27
carries. The prose below is left as authored; this header and the per-section
notes are the authority where they disagree.
| § | What it says | Status |
|---|---|---|
| The problem, measured | Neither estate has a schedule; the subsystem is built and idle | STILL TRUE (as measured 2026-08-04) |
| §1 | A schedule is “an ordinary scoped config row” at backup.schedule, loaded by ListPathAcrossScopes, writable through shc_config at global/tenant scope | SUPERSEDED. Schedules moved to the backup_schedule table under the 2026-08-07 config-vs-state ruling. ScheduleConfigPath is now a RegisterRetiredPath: writes are refused with X409073CFGRTD at Service.Set and at the daemonless overlay. Nothing reads the path any more: the loader’s carry of leftover rows was removed with SHC-3034, so the refusal is all that remains of it. |
| §1’s guard-bypass argument | ”Writing the row directly through shc_config bypasses both guards” | RESOLVED, not just argued. The bypass is closed by the retirement; that is precisely why the storage moved. The argument is correct history; it is no longer a live hazard. |
| §1’s Verdict | Retention in terraform now; schedules via CLI then a provider resource | STANDS, on the new reasoning |
| §2 Sequencing | Credential posture first | STILL PROPOSED; blocked on estate access |
| §3 the schedule set, and the 02:00 collision | The cron cadence proposal | STILL PROPOSED. The 02:00 collision it identifies is CONFIRMED and has been carried into the shipped guide. |
| §4 Retention | Pin the shipped defaults | STILL PROPOSED |
§5 shc_backup_schedule resource | The provider resource, and the daemon facts it rests on | STILL NOT BUILT. Its apiclient table is correct: BackupScheduleSet / Get / Unset / ListGet all exist and .get has no CLI spelling. Its unset-converges caveat is correct and verified. But its framing of the row as config, and its “extending shc_config with stack/environment” alternative, are SUPERSEDED: there is no config row left to extend toward. Its cron-grammar sentence is incomplete; see the note at §5. |
| §6 Runbook | What an operator runs | STILL PROPOSED; shc backup schedule list in it is a valid alias of the canonical ls |
| §7 / Open questions | What lands, what waits | STILL OPEN |
See Config versus state: the migration, whose first migration is this one, and the shipped Enable backups §7 for the operator-facing grammar and the 02:00 rule.
The problem, measured
Section titled “The problem, measured”Neither estate has a single backup schedule in any tenant. Measured read-only
on both leaders on 2026-08-04 and recorded in
docs/guide/operations/enable-backups.md
§1: shc backup schedule list returns no backup schedules on promanager
and on bitspur. shc backup ls returns five rows on promanager (every one
a pre-uninstall safety archive, every one ENCRYPTION none, every one
naming a stack that no longer exists) and no backups on bitspur.
Production data on both estates has no automated backup at all, while the
backup subsystem sits fully built: two capture engines, a seal into object
storage, a quiesce bracket with escalation, a retention pruner on a daily cron,
and a leader-only scheduler subsystem (backup_scheduler, priority 70) that
loads its schedule set on become-leader and has never had one to load.
§2 of that guide already established the cause and it has not changed: configuration is complete, the vault resolves, the buckets exist and are reachable. No schedule was ever registered. That is the whole of it.
1. The deciding question: terraform-able, or CLI-only?
Section titled “1. The deciding question: terraform-able, or CLI-only?”★ SUPERSEDED: this section’s premise was reversed within the day. A backup schedule IS a table row now:
backup_schedule, per the 2026-08-07 config-vs-state ruling (ce modules/backup/schedule_store.go). Thebackup.scheduleconfig path is RETIRED:shc config set backup.schedule=…, the-cinstall/update passthrough, the daemonless overlay and theshc_configterraform resource are all refused (X409073CFGRTD). The reachability table below is therefore a table of things that no longer work; only the “stack / environment / deployment scope: No” rows survive, and they survive for a different reason (there is no config row at any scope). Reads and deletes on the retired path still succeed, and the loader still honours a leftover legacy row so an operator can see it, let it fire, and clean it up. The section’s conclusion is unchanged; its mechanism is not.
Both, and the split matters. A backup schedule is not a table row and not an API-only object: it is an ordinary scoped config row, which is why part of this is reachable from the pipeline today and part is not.
modules/backup/scheduler.go:
const ScheduleConfigPath = "backup.schedule"The schedule’s target is not encoded in the key. It is the config row’s own
scope columns. subsystems/impl/backup_scheduler.go loads the set with
rows, err := svc.ListPathAcrossScopes(ctx, backup.ScheduleConfigPath)and ListPathAcrossScopes is WHERE path = ?, an exact path match, not a
prefix walk. So every schedule in the estate is a row at exactly
backup.schedule, distinguished from its siblings only by
(tenant_name, stack_name, environment). BackupScopeFromConfigEntry accepts
two value shapes: a bare cron string, or a map {cron, all_connected, with_dependants}. shc backup schedule set writes the map form
(modules/backup/router.go).
What that buys, and what it costs
Section titled “What that buys, and what it costs”| Reachable from terraform today | Notes | |
|---|---|---|
| Retention policy | Yes, fully | backup.retention.{daily,weekly,monthly,yearly} is ordinary scoped config; shc_config sets it at global or tenant scope |
| Backup schedule, global scope | Yes, as a bare cron string | shc_config with key = "backup.schedule" |
| Backup schedule, tenant scope | Yes, as a bare cron string | shc_config with tenant = "corp" |
| Backup schedule, stack / environment / deployment scope | No | shc_config exposes key, value, yaml, tenant and nothing else; there is no stack or environment attribute (terraform/provider/internal/provider/config_resource.go) |
all_connected / with_dependants | No | the yaml form requires every leaf to be a scalar, and would flatten to backup.schedule.cron, a different path, which the exact-match loader never sees |
There is a sharper objection than the missing scopes. Writing the row
directly through shc_config bypasses both guards the .set endpoint
applies. scheduleSetHandler calls Scheduler.RegisterScope first (which
validates the cron expression and returns X400972INVCRN on a bad one) and
Service.CheckScheduleDestination, which returns
X400077BKPSCD refusing to create a backup schedule: backup.repo (…) is configured but backup.password is not, so every scheduled archive would be written UNENCRYPTED to local disk instead of the repositoryA raw config write gets neither. A typo’d expression lands as a dead row that
never fires and never says so, and the one refusal that exists specifically to
stop a cron from writing plaintext forever is not consulted. Using
shc_config to arm a backup schedule trades away the safety mechanism this
whole subsystem’s credibility rests on.
Verdict
Section titled “Verdict”- Retention: land it in terraform now. It is ordinary config, the scopes line up, and there is no guard to bypass.
- Schedules: CLI for the first arming, then a provider resource. The
runbook in §6 is what an operator runs. The durable answer is a
shc_backup_scheduleresource (§5), which is a pure provider-side addition with no daemon change, becausepkg/apiclientalready carriesBackupScheduleSet,BackupScheduleUnset,BackupScheduleGetandBackupScheduleListGet. That is a complete CRUD surface; the provider simply does not use it yet.
Note also that neither ops repo sets any backup.* key in terraform. Both
estates’ backup.repo, backup.password and backup.aws_* values were set by
hand, out of band. Read from origin/main via the GitLab API: promanager/ops
and bitspur/ops each carry one shc_config resource, and it wires
dns.public_address only. The estate’s entire backup posture is currently
outside the pipeline that is supposed to own it.
2. Sequencing: the credential posture comes first
Section titled “2. Sequencing: the credential posture comes first”Arming a schedule before §2 is complete automates the production of compromised archives. Four separate facts force this ordering.
a. The shared S3 credential. Both hel1 estates were installed with the
same Exoscale key, and it was an Owner-role key, policy literally
{"default-service-strategy":"allow","services":{}}. Measured 2026-08-04:
either estate could list, write and delete objects in the other’s backup
bucket, and in an unrelated bitspur-k8s-archive bucket. Restic’s client-side
encryption does not help here: confidentiality is not availability, and a
delete needs no key. Backup isolation between the two estates does not exist.
Guide §9 carries the scoped-role recipe and the non-destructive probe matrix
that proves it took.
b. The restic repos must be DELETED, not rotated. restic key passwd
re-wraps the same master key under a new password. A repository whose old
password was public stays readable by anyone who ever held it, forever, after
any number of rotations. There is no restic key add verb in this product and
no key-rotation story that preserves archives. The only correct remediation for
a repo created under a published key is to destroy it and create a new one.
Two published keys are in scope. The retired constant
"shc-default-backup-password" shipped in a public AGPL repo and was the
daemon’s invented default until it was removed on ce origin/main;
docs/reference/legacy-and-breaking.md states the irreversible consequence:
archives taken under it must be treated as unencrypted. The second is
subtler: Load used to return the raw config row, so an operator who followed
the shipped guidance and set backup.password: ref+vault://backup/password
encrypted under the literal 27-byte string ref+vault://backup/password,
also a constant published in this repo, and indistinguishable from a real key
at every surface. Both estates carry backup.password = ref+vault://shc/system/backup/password, so both are in the blast radius of the
second one for any repository initialised before that fix.
The measured good news: the only repository either estate is known to
hold on a published key is the local one at /var/lib/shc/backups/, created
2026-07-27, with zero snapshots (guide §2). Deleting it costs nothing.
Whether a repository exists in either S3 bucket at all is unestablished: no
capture has ever sealed into one, and restic init runs at first capture. If
the buckets are empty, this remediation is nearly free, which is the one
advantage of having never taken a backup.
c. Existing .bak archives carry live credentials in cleartext. An
unsealed orchestrated .bak is a plain !<arch> container: manifest.json
plus an uncompressed tar, no key needed for either. It carries the deployment’s
install-time --var values verbatim and the app’s own credential files: a
Nextcloud config.php dbpassword, a frappe site_config.json
encryption_key, a postgres converter’s pg_dumpall role lines. Separately,
InstallVdeployment resolved ref+vault://…?autogen in place and persisted
the resolved value into the instance’s manifest.json, twice. ce 65789fa
fixes the mechanism (vvars stays ref-intact and the vplug data block is
rendered twice, once resolved for the jobs and once ref-intact for the sidecar),
but it does not remediate a sidecar already written, and it does not
remediate an archive already taken.
This is a hard prerequisite for bitspur specifically. bitspur/ops
declares three shc_vdeployment resources; promanager/ops declares none.
Scheduling bitspur on a daemon that predates 65789fa puts cleartext vault
credentials into every archive, on a cron, unattended. shc origin/main already
pins a ce carrying the fix (c50f7e8, 2026-08-07 09:14, which contains
65789fa at 07:11), so the prerequisite is a roll, not a code change. The
last recorded estate version is v0.0.22, which predates it; re-derive that
before acting on it.
d. Nothing prunes today, and the .bkg bundles never will. Retention has
never run against a real archive on either estate. {state_home}/exports/*.bkg
group bundles are written by decrypting members back out and are never pruned
by retention; the guide says delete them yourself.
The order
Section titled “The order”- Roll both estates to a build at or past shc
origin/main’s current ce pin. Mandatory for bitspur (three vdeployments); good hygiene for promanager. - Mint one Exoscale IAM role + API key per estate, scoped to that estate’s
own bucket. Install via
shc vault set+shc config set(guide §9). Prove the boundary with the DeleteObject / DeleteBucket probes; aDENIEDfrom a probe that never issued a request proves nothing. - Establish what is actually in each bucket. If any repository predates the
password-ref fix, empty the bucket and let the next capture
restic inita fresh repository. Do notrestic key passwd. - Delete
/var/lib/shc/backups/on both leaders: zero snapshots, published key, nothing writes to it. - Prove one manual encrypted capture end to end (guide §5.2 → §6): the row
must read
encryption=resticandlocation=restic:…, andshc backup verifymust pass. - Only then arm the schedules.
3. The proposed schedule set: needs sign-off
Section titled “3. The proposed schedule set: needs sign-off”Two constraints fall straight out of the code and both rule out the obvious answer.
The documented example collides with the retention prune.
modules/retention/crons.go sets SpecRetentionCleanup = "0 2 * * *"; the
retention cleanup cron is leader-only, daily, at 02:00. Its sweep issues
restic forget per snapshot plus one restic prune per repository, and
prune takes an exclusive repository lock. The cron expression used as the
example in shc backup schedule set --help, in guide §7, and in this project’s
own prose is "0 2 * * *". Following the documented example puts every
scheduled capture in direct lock contention with the pruner. Nothing in the
proposed set fires between 02:00 and 03:00.
A quiesce is a hard error to dependent applications, not latency. Guide §4a
is explicit: ALTER SYSTEM SET default_transaction_read_only = on does not
queue writes, it fails them, and super_read_only and Nextcloud maintenance
mode behave the same in their own vocabulary. A scheduled capture of a shared
database is, for its duration, an outage the dependent stacks see as 5xx.
Cadence and window are therefore a business decision, not a technical one,
which is why this section needs a signature.
A third consideration is specific to tenant scope. timeouts.quiesce_hold
defaults to 300 seconds; a fire that holds its freeze past the deadline
stops the stack, retakes the capture against halted containers, and starts
it again. Unattended, at night. Two stacks are candidates: openobserve
(~5 GB) and, on any tenant-scope sweep of master, traefik, which owns
:80/:443, so pausing it is a total estate ingress outage for the length of the
capture.
The shape
Section titled “The shape”ResolveScopeStacks expands a tenant-scope fire to every stack under the
tenant, and every stack in that fire shares one consistencyID, so
shc recover --consistency-id <id> restores them as a group,
DB-provider-first. Capture order within the sweep is the deployment table’s
order and is not DB-first; the consistency group, not the capture order, is
what makes the restore correct.
That gives tenant scope a property no per-stack set has: a deployment added next month is protected the day it is created, with no terraform change and nobody remembering. Protection by omission is precisely how both estates arrived at zero archives. The proposal weights that heavily.
| Scope | Target | Cron | Rationale |
|---|---|---|---|
tenant | corp | 30 3 * * * | The application tenant, where deployments get added. Tenant scope so new stacks are covered automatically. One consistency group per fire. |
stack | master / openobserve | 0 4 * * * | ~5 GB, the long one. Its own window so it cannot push a shared sweep past the hold deadline. |
stack | master / postgres | 30 4 * * * | Carries the read-only quiesce and backs Keycloak’s realms. Its own window, after the app tenant. |
stack | master / minio | 0 5 * * * | Blob store for huly; pre_backup exports its IAM database into the data volume. |
stack | master / traefik | 30 5 * * * | Ingress outage for the duration. Deliberately last, deliberately its own row, deliberately the quietest minute. |
master gets explicit per-stack rows rather than a tenant-scope row for one
reason: a tenant sweep of master would pause traefik inside a sweep whose
duration is the sum of every other stack in the tenant, and there is no
exclusion mechanism in the scope model. master holds system stacks and
changes rarely, so the “silently unprotected” risk that argues for tenant scope
in corp is much weaker there.
Open for the owner:
- Window. 03:30–05:30 UTC is a guess, not a measurement. What hours are these estates genuinely quiet, and is a short 5xx window in them acceptable?
- Cadence. Daily is proposed. Weekly halves the outage count and quadruples the worst-case loss.
traefikat all? Its ~270 MB is the ACME store. Losing it means re-issuing certificates, recoverable without a backup. Trading a nightly ingress blip for that may not be worth it.--all-connected? Not proposed. On a tenant-scope fire it is redundant, and on themasterrows it would pull peers into windows chosen for one stack.
4. Retention: pin the defaults, do not change them
Section titled “4. Retention: pin the defaults, do not change them”The shipped defaults are daily 7, weekly 4, monthly 12, yearly 5
(modules/retention/types.go), resolved per field through a Stack > Tenant >
Global cascade. They are already in force: retention needs no configuration
to run, and the cleanup cron has been firing daily at 02:00 against an empty
archive set this whole time.
The proposal is to write those same numbers into terraform rather than
change them. That makes the policy reviewable in git, immune to a default
change in a future release, and (unlike the schedule rows) it is ordinary
config with no guard to bypass, so shc_config is the correct instrument.
resource "shc_config" "backup_retention" { yaml = <<-EOF backup: retention: daily: 7 weekly: 4 monthly: 12 yearly: 5 EOF}At one capture per stack per day this keeps up to 28 archives per stack, which restic deduplicates aggressively. Do not raise these before step 4 of §2 has proven a real capture; retention deletes archives, and a pruner pointed at a repository nobody has verified is a way to lose the only copy.
One correction belongs with this: core/config/default.yaml describes
backup.retention as a “Restic forget policy” whose nulls make “restic fall
back to its built-in defaults.” Both halves are false. No restic forget
policy exists anywhere in the tree: there is no --keep-daily,
--keep-weekly, --keep-last or --keep-monthly on any call; the only
forget call site is a bare per-snapshot restic forget <id>. And a null does
not defer to restic, it selects SHC’s own 7/4/12/5. An operator choosing a
retention policy from that comment is choosing from a description of a
mechanism that does not exist. A one-hunk correction lands with this proposal.
5. shc_backup_schedule: the durable answer
Section titled “5. shc_backup_schedule: the durable answer”Still NOT BUILT, and two of its statements need correcting.
- The storage moved. This section’s
shc_configalternative is moot:backup.scheduleis a retired config path, so there is nothing to extendshc_configtoward. Going through.setis no longer merely “the entire point”; it is the only way in.- The cron-grammar sentence is incomplete.
cron.NewParser(Minute|Hour| Dom|Month|Dow|Descriptor)accepts the full descriptor set, not three:@yearly/@annually,@monthly,@weekly,@daily/@midnight,@hourly, and@every <duration>, plus an optional leadingTZ=/CRON_TZ=prefix. What the sentence gets right is the rest: 5 fields, no seconds field, no prose. A rejected expression isX400972INVCRN.- The
unset-converges caveat is CORRECT and worth keeping, verified atce modules/backup/router.go,scheduleUnsetHandler. A follower-served unset deletes the row and unregisters only its own cron; the leader keeps firing until its reload lands.- The apiclient table is CORRECT. All four calls exist, and
BackupScheduleGetreally is API-only;shc backup schedule getis not a CLI verb (the verbs areset/unset/ls, withlistas an alias).
Specified here, not built, pending sign-off on §3.
The provider exposes thirteen resources; none is backup-related. The addition is small and needs no daemon change, because the generated client already has the whole surface:
| Provider operation | apiclient call | Endpoint |
|---|---|---|
| Create / Update | BackupScheduleSet | POST /api/method/shc.backup.schedule.set |
| Read | BackupScheduleGet | POST /api/method/shc.backup.schedule.get |
| Delete | BackupScheduleUnset | POST /api/method/shc.backup.schedule.unset |
| Import / drift | BackupScheduleListGet | GET /api/method/shc.backup.schedule.list |
resource "shc_backup_schedule" "corp" { scope = "tenant" # global|tenant|environment|stack|deployment tenant = shc_tenant.corp.name cron = "30 3 * * *"}
resource "shc_backup_schedule" "openobserve" { scope = "stack" tenant = "master" environment = "default" stack = "openobserve" cron = "0 4 * * *"}Going through .set rather than shc_config is the entire point: the resource
inherits RegisterScope’s cron validation and the X400077BKPSCD refusal, so
a bad expression or a half-configured key fails at plan/apply time with a
named error instead of becoming a row that quietly never fires.
Two things to get right in the implementation. unset unregisters the live
cron only on the node serving the request; if a follower serves it, the
leader keeps firing until its own config reload, and the config-row deletion is
what makes it converge; Delete should therefore verify through
BackupScheduleListGet rather than trust the 200. And the scheduler’s parser
is stricter than shc schedule add’s: cron.NewParser(Minute|Hour|Dom| Month|Dow|Descriptor): 5-field cron and @hourly/@daily/@weekly only, no
"every 5 minutes" prose and no seconds field.
Extending shc_config with stack and environment attributes is the
tempting cheaper alternative. It should not be taken. It would widen the
raw-config write path to exactly the scopes where the guards matter most, and
the guard bypass in §1 is the reason not to.
6. Runbook: what an operator runs
Section titled “6. Runbook: what an operator runs”Estate SSH is not reachable from the box this was authored on, so none of this
was executed. Run on the leader, with the vault unlocked. Substitute the
real deployment list from shc ls --all-tenants; the “12 deployments” figure
in guide §1 is stale and bitspur has grown well past it.
# --- preflight -------------------------------------------------------------shc ls --all-tenants # derive the REAL stack list; do not assume 12shc backup schedule list # expect: no backup schedulesshc retention ls # expect: 7 / 4 / 12 / 5shc doctor run -c backupshc vault status # must be unlockedDo not proceed past here until §2 steps 0–4 are complete and one manual
capture has come back encryption=restic / location=restic:… and passed
shc backup verify.
# --- arm (order does not matter; the expressions do) ------------------------shc backup schedule set "30 3 * * *" --scope tenant -t corp
shc backup schedule set "0 4 * * *" --scope stack -t master -s openobserveshc backup schedule set "30 4 * * *" --scope stack -t master -s postgresshc backup schedule set "0 5 * * *" --scope stack -t master -s minioshc backup schedule set "30 5 * * *" --scope stack -t master -s traefik
shc backup schedule list # five rows, each with a next_run--scope stack is the default and -t/--tenant is the inherited global flag;
both are stated explicitly above because the tenant is resolved client-side
(-t → $SHC_TENANT → the persisted shc ctx -t pointer → "default") and a
schedule that lands on the wrong tenant looks identical to one that landed on
the right one.
# --- verify the FIRST fire, the next morning --------------------------------shc backup ls --all-tenants -o json | head -40Every new row must read encryption=restic and a location starting
restic:. A row reading none means the seal fell back to local plaintext;
check the logs for X200075BKPSEL (the reason) and X200079BKPLOC (the path),
and treat {state_home}/backups as secret material until it is fixed.
# --- ground truth, independent of SHC ---------------------------------------export AWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=…export RESTIC_REPOSITORY='s3:https://sos-de-fra-1.exo.io/shc-backup-<estate>-hel1'export RESTIC_PASSWORD=…restic snapshots # one per stack capturedrestic check # exclusive lock: never inside a windowshc backup ls showing completed is not proof the archive exists; guide
§6 demonstrates a deleted snapshot still listing as completed. Use
shc backup verify <id>, and restore-test one stack before trusting the set.
To back out, shc backup schedule unset --scope <same> -t <same> [-s <same>]
with the scope reproduced exactly; the scope tuple is the row’s identity.
7. What lands with this proposal, and what does not
Section titled “7. What lands with this proposal, and what does not”Lands now: neither touches an estate and neither depends on a decision:
- this document;
- the
core/config/default.yamlretention-comment correction (§4).
Blocked on owner sign-off:
- the cadence, window and retention numbers in §3 and §4;
- the
shc_backup_scheduleprovider resource (§5), whose shape follows from the scope set that gets signed off; - the
backup.tfroot in each ops repo: retention is landable immediately, the schedule rows want the resource.
Blocked on estate access: every step in §2 and §6.
Open questions
Section titled “Open questions”- Window and cadence (§3). The single decision everything else waits on.
- Is a nightly ingress blip for
traefikworth its ACME store? (§3) - What is actually in the two S3 buckets? Determines whether §2 step 2 is free or expensive. Unestablished from here.
timeouts.quiesce_hold= 300s: raise it, or accept that a stack whose capture runs long gets stopped and restarted unattended?openobserveis the live candidate.- Should the backup posture move into the ops repos at all? Today the keys are entirely out of band. Retention and schedules can be codified; the vault secrets deliberately cannot, and must never enter terraform state.
See also
Section titled “See also”- Enable backups: the manual procedure, the two engines, the read-only trap, and the per-estate credential rule
- Backup engine convergence why the destination converged and the engine did not
- Disaster recovery