Upgrading SHC
There are two distinct upgrade paths, which you should not confuse:
| What you’re upgrading | Use |
|---|---|
| A deployment (an app installed on SHC) | shc update <deployment> |
| SHC itself (the daemon binary / package) | This page |
This page is about upgrading SHC, the control plane, on one node or across a cluster, within one release line. For deployment updates, see the app update flow.
From the old line to the new line: no in-place upgrade
Section titled “From the old line to the new line: no in-place upgrade”You can’t install the new line (dev, v0.1) over the old one (main and
release/v0.0, the Swarm and Nebula line) and restart. The new line reset
its migration history: no migration file is common to the two lines. Before
any SQL runs, shcd reads its _shc_migrations ledger, finds the old line’s
rows in no catalogue it ships, and refuses to boot with X409990MIGRST
(“this store’s ledger records … no catalogue in this build ships …;
reinstall from this release, nothing migrates the store”, ce
core/db/migrate.go AssertLedgerCovered).
That refusal is the design (the
reset doctrine),
not a bug to work around.
The path is a fresh cluster on the new release, then each app’s data brought across: through the app’s own export (a database dump, the app’s export feature), or through an old-line backup archive whose manifests were converted to the new line’s shape first. The new line never reads an old-line manifest as is. The bitspur estate’s move is planned that way, app by app (SHC-1061, the plan). Which line a branch belongs to is in release lines.
Reset lines
Section titled “Reset lines”A reset line is a release whose store a cluster has to be formed from.
v0.1.14 squashed the store’s migrations into one file again (SHC-3098) and
hands the fabric out in /26 chunks. v0.1.21 squashed them again after the
store’s convention pass (SHC-3446), and is the line today. A node records the line it was formed on
as reset_line in node_bootstrap.json: shc init writes its build’s line,
shc node join the leader’s.
Nothing upgrades a node across a reset line. An estate formed before v0.1.21 is re-formed, not upgraded. On a node formed before the line:
- the package’s preinst refuses it before dpkg unpacks anything, so the
installed package and its running daemon stay as they were and
aptends with an error; - shcd, started on it anyway (a reboot), stops before anything starts and
exits 65, which
shc.servicedoes not restart.
Both print one line, X409992PRELIN, naming what the node holds from before
the line: no reset_line or another one, a fabric chunk that is not a /26,
or ledger rows of migrations the line squashed. shcd start-check asks the
same question and changes nothing.
v0.1.28 squashed the store’s migrations into one file again without a
reset, and v0.1.28 through v0.1.38 adopted a store from v0.1.21 through
v0.1.27 the first time shcd started on them. Later builds don’t: a store
that never started on one of those releases still records the files the
squash replaced, and the deb and shcd refuse it with X409992PRELIN, which
then says to start shcd once on a release from 0.1.28 through 0.1.38:
shc: [X409992PRELIN] this node was formed on this build's v0.1.21 reset line, its store's ledger records 1 migration that 0.1.28's squash replaced, 20261018013510_auto.sql; start shcd once on a release from 0.1.28 through 0.1.38, then upgrade (https://selfhostedcloud.org/guide/operations/upgrades/#reset-lines)Take such a node to one of those releases first and start shcd there once, then upgrade. A store that also holds any other row from before the line, or a node whose bootstrap or fabric chunks are off the line, is still told to re-form.
To re-form, take each app’s data out first, through its own export or a
backup the new line restores. Then form the cluster again on hosts that hold
nothing of the old one: reinstall them, or stop and remove the old cluster’s
containers and networks, since shc init refuses a host where the old
fabric’s addresses are still up (X400450ADROVL). Install this release’s
package there: its preinst refuses it on a host that still holds the old
node’s node_bootstrap.json or store. Run shc init --force on the first
node and shc node join --force <token> on every other one, install the apps
and bring their data back.
Everything below is an upgrade within one line.
Single-node upgrade
Section titled “Single-node upgrade”Within one line, a single-node upgrade is “install the new version on top of the old one and restart the daemon,” as long as the release leaves the store’s migration file as it was; one that changes it refuses the old store (see the store’s migration).
Debian package
Section titled “Debian package”cd shcgit pullmake debsudo dpkg -i ../shc_*.deb # the package restarts the daemon itselfjournalctl -u shc -f # confirm it comes back healthyThe package’s configure step restarts a running daemon once the operations
it runs have finished (shcd roll-wait, 30 minutes at most), and starts the
daemon of an initialised node that is stopped. A systemctl restart shc
after it restarts the daemon a second time. If an operation still runs after
30 minutes, the package keeps the old daemon running and names the
operation; restart shc yourself once it ends.
make install
Section titled “make install”cd shcgit pullmake installsudo systemctl restart shcWhat restart actually does
Section titled “What restart actually does”- Stops the daemon (SIGTERM → graceful shutdown of subsystems).
- New daemon process starts.
- shcd applies any pending migration from the files built into it
(ce
core/db/migrate.go); noatlasbinary runs. In a cluster the store is replicated, so only the first node that starts on a new build finds a numbered migration to apply. - Subsystems initialize in dependency order (see Daemon manager).
- Leader election runs; a new leader is elected if needed.
- Status/health reconciliation loops start; the daemon picks up running containers and resumes management.
The runtime keeps containers running during a brief daemon restart. The
independent watchdog stops containers labelled shc.single_writer if the
node’s lease expires. See the watchdog.
Multi-node cluster upgrade
Section titled “Multi-node cluster upgrade”Nodes take a new build one at a time, so for the length of a roll some nodes run the new build and some the previous one. Within one line that works: every numbered migration is additive, so the previous build runs on the migrated store, and the deployments keep serving through it (see the store’s migration). Finish the roll; don’t leave a cluster on two builds.
Rolling upgrade (recommended)
Section titled “Rolling upgrade (recommended)”Upgrade one node at a time, the nodes that don’t vote first and the leader last
(shc cluster status names the leader), and wait for each to read
healthy before moving on:
# On each node in turnsudo systemctl stop shc # or `shc node drain` to move deployments off firstsudo dpkg -i shc_<version>_amd64.deb # the package starts shc againsystemctl is-active shc # activeshc cluster status # wait for "nodes: N healthy of N" and "quorum: healthy"shc node ls # BUILD shows which nodes run the new builddpkg -i starts the daemon itself: its configure step starts an
initialised node’s daemon that is stopped, so a systemctl start shc after
it changes nothing. Skip the systemctl stop and the package restarts the
running daemon instead, once the operations it runs have finished. The first
node that starts on a build carrying a numbered migration applies it, once,
and every node after it finds it applied.
The terraform provider rolls in the same order when package.version
changes, gates each node on shc node ls, quorum and shc doctor run, and
stops at the first node that fails. See
the provider reference.
Once every node runs the new build, the leader finishes each deployment’s
stored render again with what the new build computes, and rolls only the
replicas whose documents changed, a few deployments at a time. Nothing
needs redeploying. shc doctor run shows it under app.refinish: the
nodes it still waits for, what of a node the new leader cannot read yet
(its runtime or its fabric address, for a minute or so after it starts),
and any deployment it could not re-finish.
Two important concerns during rolling upgrade:
- Leader continuity. The leader may be the node you’re upgrading. Raft will elect a new leader automatically; in-flight leader-only operations (scheduled job ticks, retention pruning) may be paused until election settles (typically < 5s).
- Raft quorum. Don’t stop a majority at once. With 3 nodes,
never have more than 1 offline. With 5 nodes, at most 2. The
shc cluster statusoutput shows quorum health.
Draining before upgrade (stacks migrate off)
Section titled “Draining before upgrade (stacks migrate off)”For a cleaner upgrade where the node takes no user traffic while being restarted:
shc node drain <node-id># Wait for stacks to reschedule (or relocate them with `shc update --where`).sudo systemctl stop shcsudo dpkg -i shc_<version>_amd64.deb # the package starts shc againshc node uncordon <node-id>Drain asks SHC’s placement engine for eligible targets and reports each
relocation. A deployment with no eligible target refuses the drain unless
--force permits stopping it in place. Check the drain result before
stopping shcd.
shc node uncordon returns the node to service from any state. Nothing
moves back: the node takes work as capacity is needed. To stop a drain
and keep the node closed, shc node undrain <node-id> leaves it cordoned
and says how many deployments had already moved.
The store’s migration
Section titled “The store’s migration”Until v1, SHC’s store is built by one generated migration file
(SHC-2197), the ce module’s migrations/, shipped inside shcd with
its atlas.sum and tracked in the _shc_migrations ledger. Behavior:
- A fresh store is built from the file on the first start.
- A store made from the same file starts as it is. Nothing applies.
- A store made from another file is refused at start, before any SQL
runs:
X409990MIGRSTwhen the store’s file is older than the build’s (“reinstall from this release, nothing migrates the store”),X409991MIGNEWwhen it is newer. Until a release ships the file, a schema change regenerates it, and that is no in-place upgrade: install the new build on a fresh cluster and bring each app’s data back from its backups. - Once a release ships the file it is frozen, and a schema change is a numbered migration on top of it. The first node that starts on the new build applies it once, as one atomic batch the store’s leader executes, and the cluster keeps working while the other nodes still run the previous build.
- Every numbered migration is additive (a new table, a new nullable or
defaulted column, a new index), so a node still on the previous build
runs on the new schema for the length of a roll. Removing or reshaping
what the previous build reads takes two releases: the first stops
reading it, the second removes it and names the first in a
-- shc:last-reader-removed-in vX.Y.Zline. - Down migrations do not exist. The loader skips
.down.sqloutright. Rollback strategy is “backup first, forward-only in production.” - In cluster mode DDL is raft-replicated, so the file the first node applies builds the store for every node. Wiping and rejoining a node replays it rather than escaping it.
- A failure applying the file aborts daemon startup. The daemon fails fast rather than running against an inconsistent schema, and the log shows the error.
- A node formed before the build’s reset line logs the one
X409992PRELINline and exits 65 before any subsystem starts, and the package’s preinst refuses it first (see Reset lines). - An older build on a store a newer build migrated logs the one
X409991MIGNEWline and exits 78 before any subsystem starts, so the store engine, the vault and the container runtime never come up on it.shc.servicedoes not restart it on that code: the node stays down until it runs the newer build.
Never upgrade a cluster past a failed start on one node.
Pre-upgrade backups
Section titled “Pre-upgrade backups”For a production cluster, confirm the control-plane snapshot is current
before a major upgrade. There is no operator verb that takes one on
demand: the leader-only db_snapshot_cron writes a consistent dump of the
control-plane database to {state}/snapshots/control-plane/ every 24
hours (raft-consistent in cluster mode, VACUUM INTO single-node), and
that dump is the operator-level recovery artifact. Check its timestamp on
the leader before you upgrade:
shc db snapshot ls # the newest nightly, and any taken on demandshc db snapshot create # take one now, on the leaderIf the newest snapshot is older than the change you are about to make,
take one with shc db snapshot create before proceeding. Retention is
maintenance.control_plane_snapshot_retention (default 7) for each kind
on its own, so an on-demand snapshot never prunes a nightly. This is
control-plane state, not app data; take stack backups separately.
See the backup module for
the full backup model and
the backup/restore flow
for stack-level backups.
Compatibility policy
Section titled “Compatibility policy”- Patch (
x.y.Z): safe to roll forward in any order. No schema or wire-format changes. - Minor (
x.Y.z): may add columns, endpoints, or events. Older clients still work; mixed-version clusters should not run for long. - Major (
X.y.z): may break CLI/API/config compatibility. Read the release notes before upgrading.
Check the current version before and after:
shc versionshc node ls # the BUILD column is the shcd each node runsRollback
Section titled “Rollback”See Rolling a daemon back for the full procedure. The summary:
- Decide whether the roll crossed a migration.
git diff OLD..NEW -- go.mod go.sumin theshccheckout: empty means thecepin did not move, the migration file did not change, and the roll is reversible by binary alone. - If it did not: install the old deb node by node, primary last, waiting for
each daemon to answer before the next.
dpkg -iaccepts an older version silently. - If it did: restore the control-plane snapshot first, then install the old deb. There are no down-migrations, and none can be written after the fact.
Two things to know before you rely on any of this:
dpkgdoes not keep the deb it replaced. A release’s deb can be fetched again, fromhttps://downloads.selfhostedcloud.org/shc/<ver>/or withapt install shc=1:<ver>on a node that reads the Based apt repo. A snapshot build is in neither, so save the outgoing deb before you roll or your rollback artifact will not exist.- A daemon downgraded across a migration refuses to boot. Its ledger
holds a migration file newer than any the older build ships, so shcd stops
with
X409991MIGNEWand exits 78 before any subsystem starts (cecore/db/ledger_newer.go, read from the node’s own copy of the store). That’s why step 3 restores the snapshot first: the snapshot’s ledger predates the migration.
See also
Section titled “See also”- App update flow: upgrading a deployment, not SHC
- Release lines: the old line and the new line
- Daemon manager: what happens at startup
- Cluster module: cluster state and recovery
- Installation: pick an install method