Lifecycle hooks
Layer 11. Lifecycle. Lifecycle
Hooks let an app run jobs at well-defined points in the SHC
orchestration pipeline, e.g. initialize a database after
pre_install, dump it before a backup, notify an external system
after install completes.
Source of truth:
modules/app/hooks.go
(the HookExecutor),
modules/job/types.go
(the JobDefinition schema),
modules/job/service.go
(JobLoader, JobExecutor).
For where exactly each hook fires in the pipeline, see the flow docs: install · upgrade · uninstall · backup/restore.
The 10 hooks
Section titled “The 10 hooks”| Hook | Fires at |
|---|---|
pre_install | After variables are merged, before compose up (install flow) |
post_install | After every replica of the deployment reads ready (Readiness comes first), before the A200100STKINS deployment-installed event |
pre_update | Before re-rendering compose during update |
post_update | After every replica of the deployment runs the new config and reads ready (Readiness comes first) |
pre_uninstall | Before compose down (uninstall step 6) |
post_uninstall | After compose down and network removal |
pre_backup | Before volumes are rsync’d into staging |
post_backup | After restic snapshot completes |
pre_restore | Before extracting the backup into target deployment volumes |
post_restore | After volumes are restored and converters applied, with the deployment running. On a clone or move target that is once the target is brought up, and a failure there fails the clone or move |
All hook names are checked against hookNames in
hooks.go; unknown names are
ignored.
Readiness comes first
Section titled “Readiness comes first”post_install does not fire when the containers have started. It fires
once every replica of the deployment reads ready: each service is
running and, where it has a healthcheck, healthy, and a restart “no”
service has ended 0. An install whose replica never reads ready ends
X503764PODWAI, is rolled back, and never runs
its post_install hook.
So a service’s readiness must never wait on something a post_install
job delivers. The hook waits on the service and the service waits on the
hook, and the install rolls back at the 15-minute readiness limit. A
start_period does not help: a container inside its start period reads
starting, which is not ready.
post_update keeps the same contract. An update’s replicas reach their
nodes through the rollout once the update’s spec is committed, so an
update with a post_update chain commits, waits until every replica runs
the new config and reads ready, and fires the chain only then. Its jobs
therefore run in the new containers, wherever they are placed. An update
whose replicas do not all read ready within the same 15 minutes never
runs the chain and fails with X500908HKFAIL.
An lxc start the node refuses counts as refused at every start only once
the node refuses it again, so a start that fails once and starts on the
node’s retry within the window does not fail the update. An update that
fails its readiness gate keeps its post_update chain pending, which the
deployment’s next shc update (its plan names the chain) or shc start runs.
An update with no post_update chain returns without waiting for the
rollout.
shc install --no-wait does not skip this wait for an app with a
post_install chain: the chain still fires only once every replica reads
ready, and the install still ends X503764PODWAI when one never does. For
an app with no post_install chain, --no-wait returns once the replicas
are handed to their nodes. A replica that fails to start afterwards is not
rolled back; shc status shows it. A replica the install’s own node could
not start still fails the install at once.
A service that needs something only the hook can produce (a binary from an app store, a minted key) keeps waiting for it instead of exiting, and its healthcheck passes while it waits:
- Nextcloud’s
notify-pushwaits for the binarynextcloud-initinstalls. Its probe passes while the binary is absent and is the listener’s answer once it exists (apps/flagship/nextcloud/compose.yaml). mailbox-reconcilerreads healthy without the API key its seal hook delivers.
Two tests hold the two sides. tests/_ce/modules/app/post_hook_after_ready_test.go
fails if ce runs post_install before the replicas are up or for an
install that ended waiting.
tests/unit/catalog/nextcloud_notify_push_idle_test.go fails if
notify-push stops reading healthy without its binary, or gives up
waiting for it.
An update of a stopped deployment leaves its post hooks for the next start
Section titled “An update of a stopped deployment leaves its post hooks for the next start”shc stop then shc update is a normal window step: lock the app,
change its config, start it. The update renders and applies the new
config and leaves the deployment stopped. It does not run the part of
its post_update phase that needs a service the operator stopped,
because nothing could exec into it:
- the
post_updatechain, when one of its actions needs a stopped service: acontext: servicejob on it, astartorstopof it, or anhttpjob or a templated service when every service is stopped (acontext: containerjob needs none); - the consumer plugs’
jobs.post_deploymentafter the chain, when the chain waits or every service is stopped.
The update says so on its result line with
X200121HKDEFR, naming the jobs and
shc start <stack>, and records them. The next operator shc start
(or shc start -S of the services they need) runs them once the
deployment reads ready, keeps the values they mint as an update does,
and reports them under its result with X200122HKDRAN. They run once.
A start that leaves a service they need stopped runs none and says
X200124HKDWAI. A job that fails, or one that changes a value its
services read when they were rendered, fails the start’s answer with
X500908HKFAIL naming shc update; the services stay up. An update
whose chain runs forgets anything an earlier update left.
Two things cannot wait for a start, so the update refuses them before it
changes anything: a plug vapp: instance it would make in a deployment
whose every service is stopped (X409912HKNDEF), and an update that
cannot read which services the operator stopped (X503916HKSTOP).
tests/_ce/modules/app/deferred_hooks_6993_test.go holds fizzy’s and
librechat’s shapes.
How a hook is defined
Section titled “How a hook is defined”Hooks are declared in app.yaml as a map from hook name to an ordered
list of steps. A step is a job name (not an inline command) or a
stop: / start: list of service names. Job names refer to job
definitions loaded from *.job.yaml files in the same app directory.
stop: stops those services so a one-off job can own their data.
start: starts them again. Neither changes a replica count; replicas
come from --scale on install and update. Where the process runs
replica projects, stop and start go through that process’s pod runner.
Otherwise they are compose stop / compose start of the named
services on the deployment’s project. A hook run that fails starts
again every service it stopped and did not start. A run that finishes,
or a job that emits ::shc-hook:halt::, does not start those services again.
The scale: hook action is gone; a step that names it is refused.
hooks: post_install: - init-db - notify-admin pre_update: - stage - stop: [db] - migrate - start: [db] pre_backup: - flush-cache pre_restore: - drop-existing-schemaname: init-dbcontext: serviceservice: postgrescommand: | psql -U {{ vars.admin_user }} -c "CREATE EXTENSION IF NOT EXISTS pg_trgm;"timeout: 60on_failure: abortJobs run sequentially in the order they’re listed. One job fails → the hook is considered failed, but see “failure behaviour” below.
A job that seeds its datastore
Section titled “A job that seeds its datastore”A job that drops or replaces whatever schema its datastore holds, a
first-boot seed ceremony, says so with seeds: true. In post_install
and post_update, SHC refuses that job while the deployment, or a
provider it binds, carries a standing restore seal, so the install that
follows a restore never wipes it. shc backup clear-restore-seal lifts
the seal once the restored data has been checked. A job whose command
drops a table or runs a framework’s destructive migration is refused the
same way without the declaration.
name: app-initcontext: serviceservice: appseeds: truecommand: php bin/console app:install --forceA job that restarts a service it reconfigured
Section titled “A job that restarts a service it reconfigured”A service that reads a file only at start keeps running the old content
when a job rewrites that file. The job asks for the restart itself by
printing ::shc-hook:restart:<service>:: on a line of its own, and
restarts: on the job names every service it may restart. SHC stops and
starts each named service once, after the job ended 0, the same way a
stop: and start: step would. A job that changed nothing prints no
marker and restarts nothing. A failed job restarts nothing. A marker for
a service restarts: doesn’t declare fails the job with X500127HKRSTU
before anything restarts. The rule holds for *.job.yaml hook jobs and
for a plug’s jobs.post_deployment jobs alike, and the catalogue test
TestEveryHookRestartIsDeclared holds every printed marker to its job’s
restarts:. SHC reads restarts: from 0.1.28, and 0.1.27 and older
refuse the field at install and update, so a package that uses the marker
declares requires.shc: "0.1.28" or later in app.yaml. The catalogue
check fails a package that uses it without that.
Socket jobs.integration jobs may also declare restarts: and print the
marker since 0.1.29. They restart services of the provider deployment,
after the job succeeds and before the next integration job runs. A socket
job that declares restarts: requires requires.shc: "0.1.29" or later,
even if it prints no marker on a replay: 0.1.28 and older refuse the field
with X400361INTFLD. TestEveryHookRestartIsDeclared checks socket markers
against the job’s declaration and the provider app’s compose services too.
jobs: post_deployment: - name: rewrite-config context: service service: app restarts: [app] command: | render-config > /etc/app/config.json.next if cmp -s /etc/app/config.json.next /etc/app/config.json; then rm -f /etc/app/config.json.next else mv -f /etc/app/config.json.next /etc/app/config.json echo "::shc-hook:restart:app::" finetbird’s netbird-rewrite-management-json is the shipped case. It asks
for the restart when management.json is newer than the management
process, because compose’s init container can write the new file before
the job runs.
Job contexts
Section titled “Job contexts”A job’s context field selects where it runs. The available
contexts (from JobContext in
job/types.go):
| Context | Runs in | Typical use |
|---|---|---|
service | An exec on an already-running compose service | Seed a database after post_install |
container | A new container that runs once, of its image, or with no image of its service’s own definition | Run migrations in a throwaway container (pre_install before the deployment is up) |
http | A host-side HTTP request to url | Ping a webhook, hit an admin API |
graphql | A host-side GraphQL mutation against url | Call a GraphQL admin endpoint |
pipeline | A sequence of steps (each a sub-job) | Multi-step provisioning |
playwright | A Playwright browser script against url | Automate a browser-based setup UI |
Container jobs per runtime
Section titled “Container jobs per runtime”A service job runs on every runtime. A container job depends on the
runtime of the node that runs it:
| Runtime | container job |
|---|---|
docker, podman | compose run --rm in the replica’s project. On podman the job starts through SHC’s exec helper, as a service does, so it holds docker’s supplementary groups and default capabilities and becomes its user: the way docker does. Its image is pulled first when the node lacks it. |
incus | A project of its own, shcjob-<run>-<job>, holding the one job. The incus executor brings it up, reads it back until it has ended and removes it with its project, whatever the outcome. The job’s output is the instance’s console log, so stdout and stderr arrive as one stream. container_name is not carried over: incus names its own instances. |
apple | A project of its own, as on incus, whose one container starts held. Its output is the container’s log. |
Why apple holds the job: a job on the fabric runs at a pin the node’s DNS
and policy already know. apple assigns a container’s address itself as it
starts the container and refuses a pinned one (ipv4_address is a refused
key on apple, see compose keys). So the job’s first
process waits before its command until the address apple gave it is the
job’s pin, and only then runs. A container that holds no address after 30
seconds fails the job with X500359FABPIN before its command runs.
A container job’s volumes
Section titled “A container job’s volumes”A job’s volumes go through the mount policy a service’s do. A named volume becomes the deployment’s data bind, and the fields it names (handover:/srv/handover:0750:1000:1000, noinit) are applied on the node that runs the job before its container starts, as they are for a service’s mount (compose keys): while the directory is new or empty, on the directory itself. A job copies nothing of its image into a volume. A mount that names no fields is made if it is missing and left as it is.
A container job that runs as its service
Section titled “A container job that runs as its service”A container job that names a service and no image runs as one new
container of that service’s own definition, as compose run --rm --no-deps
makes one, with the job’s command. Nothing of the service is written
twice:
name: medusa-migrationcontext: containerservice: medusaafter: [valkey]before: [medusa, medusa-worker]command: ["npx", "medusa", "db:migrate"]- It takes the service’s image, environment and
env_file, volumes and binds, configs, secrets,user,working_dir, capabilities and resource limits, as the replica’s document holds them. - It takes none of its ports,
healthcheck,restart,depends_on,container_name, labels or network aliases: a job never answers for the service. It joins the replica’s networks as every container job does, at a fabric address of its own. - The job gives the
command, and may give anentrypoint, auser, aworking_dirandenvironment, whose keys win over the service’s. An exec-formcommandruns under the service’s ownentrypoint, as compose run does. It names novolumes: a job that needs another mount names itsimageinstead. - It runs on the node that holds the service’s lowest replica, whose mounts
and networks are made ready while that replica stays stopped, so the
service need not run: a migration names it in
before:. A scaled service runs the job once, from replica 0. For a group member name the member (postgres-1), never the group.
Allowed contexts per hook
Section titled “Allowed contexts per hook”Not every context is valid for every hook: pre_install can’t
exec into services that don’t exist yet. The mapping is defined
in allowedContexts in
hooks.go:
| Hook | Allowed contexts |
|---|---|
pre_install | http, container |
post_install | http, container, service |
pre_update | http, container, service |
post_update | http, container, service |
pre_uninstall | http, container, service |
post_uninstall | http, container |
pre_backup | http, container, service |
post_backup | http, container, service |
pre_restore | http, container, service |
post_restore | http, container, service |
Jobs using a disallowed context are skipped with a warning
(X500908HKFAIL): the hook is not aborted, the invalid job simply
doesn’t run. The helper _validate_job_context enforces this.
Why service is disallowed in pre_install and post_uninstall:
at pre_install the deployment hasn’t started yet, so there’s no
container to exec into; at post_uninstall the containers are gone.
pre_restore runs while the deployment is still up, before the restore
stops it, so a service job can ask the running app whether the restore
may go ahead. A job that prints ::shc-hook:refuse:: <why> and exits
non-zero stops the restore before anything is stopped, backed up or
written (X409876RSTHKR, carrying its line); one that fails without the
marker is a warning and the restore goes on. The hooks fire from the
deployment’s stored package on a node that holds its replica and no app
tree. A restore into a target that does not run, a recover onto an absent
target or the target of a clone or move, fires no pre_restore: nothing
runs for it to ask. Its post_restore runs once the replicas start.
Variables available to hook jobs
Section titled “Variables available to hook jobs”The HookExecutor assembles a variable dict before handing the job
to JobExecutor:
vars.yaml: the merged variable dict persisted during install (see the install flow). Exposed to templates asvars.*.- Secret references resolved: any
ref+vault://...strings invars.yamlare resolved before the job runs, so commands see real values, not URIs. This is specifically so bootstrap hooks (e.g. hashing an admin password) see the real password. SeeloadDeploymentVarsinhooks.go. Secret resolution usesstrict=False: an unresolvable ref leaves the URI literal in place rather than aborting. - Job-level constants:
stack_name,tenant_name,environment,node_id,deployment_dirpassed through theJobExecutorconstructor.
In job YAML, reference them with Jinja:
command: | echo "installing {{ stack_name }} in {{ environment }}" psql -U {{ vars.admin_user }} -c "SELECT 1;"on_failure and the hook failure model
Section titled “on_failure and the hook failure model”Each job has an on_failure field (see OnFailure enum). But note
the hook-level contract is independent:
At the job level
Section titled “At the job level”on_failure: abort: if the job fails, abort the hook (subsequent jobs in the same hook list don’t run).on_failure: continue: log the failure, run the next job anyway.
At the hook level
Section titled “At the hook level”A hook failure (a job exiting non-zero, or any other error inside
the hook chain) is logged as a warning (X500908HKFAIL) and
does not propagate to the caller.
That means: the pipeline treats hooks as best-effort, with exactly
one exception. A failed post_install hook does not fail the
install. A failed pre_backup hook does not fail the backup.
The one exception: pre_update gates the update
Section titled “The one exception: pre_update gates the update”pre_update is the phase that runs while the old containers are
still serving, which makes it the only place an app can answer “may
this version replace the one on disk?”. A refusal there is therefore
fatal to the update: the pipeline abandons it with
X409005HKGATE before anything is pulled,
recreated or scaled, and the deployment keeps serving its previous
version.
What is and is not restored:
- Containers: untouched. Nothing was brought down or up.
- Rendered artifacts: the deployment dir holds the NEW
compose.yaml,app/andvars.yaml, the same as after any other failure between the render and compose-up. Re-run the update once the hook’s complaint is answered; nothing needs repairing first. - The manifest sidecar: still records the version the deployment is running, because it is written only after a successful compose-up.
A context: service pre_update job is SKIPPED, not failed, when
its service has no running container (X200912HKSKIP). docker compose exec into a stopped service cannot succeed, and the update
path is also how a deliberately stopped deployment is re-rendered:
shc stop then shc update, and both legs of a move’s fast path,
whose stop precedes the re-render and its rollback. A gate that must
hold with the deployment down belongs in a context: container job (those
still gate) or in compose itself.
Belt and braces is still the recommendation
Section titled “Belt and braces is still the recommendation”Two of the three packagings that gate an update (postgres, the
major-version ladder, and jellyfin, the one-way library migration)
keep a second copy of their refusal as a compose service whose
exit code the app’s depends_on: service_completed_successfully reads. Keep doing that for anything
irreversible: the compose copy covers the stopped-deployment window the
service-context hook skips, and it fires on every start, not only on
the ones SHC drove as an update.
Other phases have no gate. If pre_install is expected to provision
something the container depends on, write the container to retry or
refuse to start until the precondition is met. Don’t rely on the
hook failing the install.
Jobs that report a value: read_only and sensitive_output
Section titled “Jobs that report a value: read_only and sensitive_output”A job no hook names runs when something asks for it: shc job run <job>, a
schedule, or the shc_job_output data source of the Terraform provider. Two keys
say what such a job does.
read_only: true says the job changes nothing. shc_job_output can start a job
at plan time, so it refuses one that doesn’t declare this. SHC doesn’t inspect
the command: the declaration is your promise.
sensitive_output: true says the job prints a secret. Its stdout, stderr and
error are sealed under the vault’s key on its run’s row, and reading them takes
stack:manage on the deployment. stack:read still sees that the run happened
and how it ended, and stack:operate can start it without learning its answer.
name: list-projectscontext: serviceservice: webread_only: truesensitive_output: truecommand: ./list-projectsA job that declares sensitive_output has to run on demand in service context.
A container job’s stdout is its container’s log, which the collector ships and
stack:read reads, so it’s refused with X400028JOBSNS. A hook or start job’s
output reaches its caller and the daemon’s log through the failure text, so it’s
refused with X400029HKSENS. With the vault locked, the run is refused before it
starts with X503919JOBSEL.
Job file loading
Section titled “Job file loading”JobLoader scans the app directory
(paths.app_dir(tenant, stack, environment)) for files matching
*.job.yaml. Each file is parsed as a single JobDefinition. The
lookup by job name is linear but cached per hook invocation.
Job files are copied into the deployment during the install flow alongside the rest of the app sources.
Start jobs: after and before
Section titled “Start jobs: after and before”Run-once work is a job, never a service. A job file that sets after:,
before: or both is a start job. It orders the deployment’s start, and no
hook names it.
| Field | Value | Meaning |
|---|---|---|
after | list of service or start job names | The job runs once a replica of each service reads ready (running, and healthy where it has a healthcheck) and each job named ended 0. |
before | list of service or start job names, a service optionally as name:stopped | Those services start only once this job ended 0. A job named here runs after this one. On an update, a service named name:stopped has its old replicas stopped before the job runs. |
name: seedcontext: serviceservice: counterafter: [counter]command: redis-cli setnx visits 0name: migratecontext: containerimage: myapp:{{ vars.version }}after: [db]before: [web]command: ["python", "migrate.py"]name: checkpointcontext: containerservice: mediabefore: [media:stopped]command: ["media-server", "--checkpoint-library"]What SHC does with them:
- An install and each update run every start job once for the deployment, never once per replica. A service job execs into one replica of its service, on whatever node holds it.
- A replica of a service a job’s
beforenames waits, on any node, until that job’s run ended 0. The order is the same one SHC keeps between pods fordepends_on. - A job that does not end 0 fails the install, which rolls back. An update runs its start jobs before anything of it starts, against the replicas already running, and one that fails ends the update before its spec commits.
- A job runs as long as its own
timeoutlets it, counted from its command’s start (600 s when unset; for anhttpjob, its attempts and the pauses between them). A container job’s bring-up (its image, its container’s create and start) has its own bound: 15 minutes, or thetimeoutwhen longer. Ten more minutes cover reaching its node and answering. The 15-minute settle window bounds how long an install or update waits for replicas to read ready; it does not count the time a start job runs, nor a replica that waits for a running job’sbefore. A job still running past its own budget fails the install, naming the job. - A job whose
afternames a service the same update adds, or that runs after such a job, runs as an install runs it. The update commits its spec, the new service starts, and the update waits until the service reads ready and then runs the job. If the job fails, the update fails after its commit, and the services itsbeforenames do not start. - A job whose
beforenames a service asname:stoppedis for work that must not run beside the old server, such as a checkpoint or an archive of its data. On an update SHC stops that service’s old replicas, runs the job once they read stopped, and starts the service on the new version only after the job ended 0. If the job fails, the old replicas start again as they were and the update fails before its spec commits. On an install there is no old replica, so the job runs as any start job. A service its operator stopped stays stopped. - A move and a failover keep the run the deployment has, so nothing runs again. A restore keeps it while the cluster still holds the deployment’s spec, and runs the jobs on a cluster that does not. A restart never runs one.
The install refuses a start job with a context other than service,
container or http, a service job whose after does not name its own
service, a container job with neither image nor service, one that runs
as its service and names volumes, a job named apart from its file, a
before entry that is neither a name nor name:stopped, a start job
written name:stopped, a name the app does not have (a service a job
runs as included), a job named like a service, a start job a hook also
names, and an order that
never begins (a job before what it runs after, directly or through
depends_on). The error names the job that failed. shc validate checks
the names.
Common patterns
Section titled “Common patterns”Bootstrap a database after install
Section titled “Bootstrap a database after install”hooks: post_install: - create-schemaname: create-schemacontext: serviceservice: postgrescommand: | psql -U {{ vars.admin_user }} -d {{ vars.db_name }} -c " CREATE TABLE IF NOT EXISTS users (id SERIAL PRIMARY KEY); "timeout: 60on_failure: abortRun a migration in a throwaway container before the deployment starts
Section titled “Run a migration in a throwaway container before the deployment starts”hooks: pre_install: - migratename: migratecontext: containerimage: myapp:{{ vars.version }}entrypoint: ["python", "migrate.py"]networks: - shc_{{ tenant_name }}volumes: - "{{ deployment_dir }}/data:/data"timeout: 300on_failure: abortNotify an external system via webhook
Section titled “Notify an external system via webhook”name: notifycontext: httpurl: https://hooks.example.com/deploymethod: POSTheaders: Authorization: "Bearer {{ vars.webhook_token }}"body: stack: "{{ stack_name }}" tenant: "{{ tenant_name }}" event: "installed"max_retries: 3retry_interval: 10Flush a cache before backup so the snapshot is consistent
Section titled “Flush a cache before backup so the snapshot is consistent”hooks: pre_backup: - flush-cachename: flush-cachecontext: containerimage: redis:7entrypoint: ["redis-cli", "-h", "redis", "BGSAVE"]networks: - shc_{{ tenant_name }}timeout: 30on_failure: continueon_failure: continue because a cache that can’t flush is not worse
than not flushing it; the backup should still proceed.
Debugging
Section titled “Debugging”Where to look when a hook didn’t run
Section titled “Where to look when a hook didn’t run”- Check the daemon log for
X500908HKFAIL: that’s the hook failure event class. shc logs -s <stack>: the container-context job’s output.- The
JobRunModeltable has one row per executed job:shc job history -s <stack>shows recent runs. - For
service-context jobs,docker logson the service container will show theexec’d command’s output.
Hook silently skipped
Section titled “Hook silently skipped”Most common cause: a job is using a context the hook doesn’t allow
(e.g. pre_install with context: service). Check for
X500908HKFAIL with context 'X' is not allowed for hook 'Y' in the
message.
Variables not resolved
Section titled “Variables not resolved”If {{ vars.admin_password }} appears literally in a job’s command,
the variable isn’t in vars.yaml (typo or missing from the merged
vars). If a ref+vault://... URI appears literally, the vault
provider couldn’t resolve it (see vault provider logs).
Related
Section titled “Related”- App definition: the full
app.yamlschema - Plugs & sockets: integrations (plugs/sockets) also use jobs (unintegration jobs)
- Flow: install: where
pre_install/post_installfire - Flow: upgrade:
pre_update/post_update - Flow: uninstall:
pre_uninstall/post_uninstall - Flow: backup/restore:
pre_backup/post_backup/pre_restore/post_restore - Architecture: job module: the
JobExecutorinternals - Error reference:
X500908HKFAIL: the hook-failure event