Common issues
Detailed, symptom-first walkthroughs. For quick symptom→page routing see the troubleshooting index. For every error code’s exact definition see the error reference.
The commands below run in a tenant. Select yours once with
shc ctx -t <tenant>, or add -t <tenant> to each command.
Install failures
Section titled “Install failures”Symptom: X503738DMNREQ, shc install requires shcd
Section titled “Symptom: X503738DMNREQ, shc install requires shcd”shc install, shc update and shc uninstall send their work to shcd,
which renders the deployment and keeps its state. When the CLI cannot reach
shcd they refuse before anything changes; there is no local fallback
(SHC-1328, ce modules/app/commands/install.go). Start it with
systemctl start shc, or on a Mac with
sudo launchctl kickstart system/org.selfhostedcloud.shcd, run
shc doctor, and if it will not stay up read
Daemon won’t start.
Symptom: X504740DMNTKN, shcd took the request and did not answer in time
Section titled “Symptom: X504740DMNTKN, shcd took the request and did not answer in time”shcd took the request of shc install, shc update or shc uninstall
and gave no answer within the CLI’s wait, which is 30 seconds. It goes on
with what it took, so the deployment may be coming up, may have failed, or
may never have been recorded. Read it with the shc status command the
error names before you run the verb again. A node far from the leader or
under heavy load is the usual cause (SHC-2159).
Symptom: X400976TNTNON, no tenant named
Section titled “Symptom: X400976TNTNON, no tenant named”The command named no tenant, and nothing picks one for you. Name it with
-t, $SHC_TENANT or shc ctx -t. A cluster made through the portal has
default; one you installed yourself has only master until you add one
(Tenants):
shc tenant ls # the tenants this cluster hasshc ctx -t default # select it for every later commandshc install <app> -s <name>Symptom: the app installed but its address shows 404
Section titled “Symptom: the app installed but its address shows 404”shc install succeeded and the app is healthy, but a browser gets Traefik’s
404 and shc ingress ls shows no route for it. An app gets a web address
only from --expose <host>; SHC never invents one. The install says so
with warning X200951HSTNOA. Give the installed app its address with
shc update:
shc update <stack> --expose <stack>.example.comshc ingress ls # the route is there nowSymptom: X400903INVAPP (“invalid app”)
Section titled “Symptom: X400903INVAPP (“invalid app”)”Install aborted validating the app definition. Causes, in rough order of likelihood:
- Required variable not provided. The app declares
vars.admin_password@secret!but you didn’t pass--var admin_password=.... Check which fields are!(required) in the app’svars:block orvars.yaml. - Required plug has no viable socket. An app with
required: trueon a plug refuses to install unless a matching socket exists. Either install the provider stack first, or pass-i <provider>. - Integration spec doesn’t resolve.
Error detail says “no matching sockets found” or
“multiple plugs match”: use the explicit
-i plug=provider.socketform. See plugs-and-sockets: CLI spec. - Malformed
app.yamlorvars.yaml. YAML parse error or a field of the wrong type. Runshc validate <path-to-app-dir>to get a precise schema error.
Symptom: X409003STKEXS (“stack already exists”)
Section titled “Symptom: X409003STKEXS (“stack already exists”)”A stack with that (tenant, stack, environment) tuple is already
installed.
shc ls --all-tenants # find the conflicting stackshc uninstall <stack> -e <env> # remove it, OR…shc install <app> -s <name> -e <env> # pick a different stack/envSymptom: Stack starts but immediately fails
Section titled “Symptom: Stack starts but immediately fails”Install reached compose up but containers exit. The install flow
waits for a healthcheck window (configurable per-app via
timeouts.start) and then rolls back.
shc logs -s <stack>: raw container logsshc events --follow: daemon lifecycle eventsshc events: structured events- Check
X500907STKFALdetail in the logs; it includes which service’s healthcheck failed
Common root causes:
- Missing integration variable: template rendered
DATABASE_URL=postgres://user:@host/db(empty password) → provider integration hadn’t finished when compose started. Re-run with-i <provider>to wire the integration, or install the provider first. - Image architecture mismatch: running on arm64 but the
app’s image is amd64-only.
Check
requires.architecturesinapp.yamland the image’s manifest. - Port already in use: something else on the host is bound to a
public_portstarget.ss -tlnp | grep <port>. - Volume mount failed: the storage provider refused the mount
path; check
shc doctor run --category storage.
Symptom: Install hangs on “pulling image”
Section titled “Symptom: Install hangs on “pulling image””Docker pull is slow or unreachable from the target node.
# On the target node:docker pull <image> # reproduce directlyshc node ls -v # find which node it targetsshc inspect deployment <stack> # see assignmentIf pulling works manually but SHC’s pull hangs, the issue is usually Docker daemon health. Restart Docker on the node.
Symptom: X404002APPNOF (“app not found”)
Section titled “Symptom: X404002APPNOF (“app not found”)”The app name isn’t in any registered repo.
shc repo ls # check repos are configuredshc repo sync # force-refreshshc search <partial-name> # confirm the app existsIf it’s a local file path repo, make sure shc has read access to
the directory.
Integration issues
Section titled “Integration issues”Symptom: X400903INVAPP (“no matching sockets found”)
Section titled “Symptom: X400903INVAPP (“no matching sockets found”)”Auto-match failed because the consumer’s plugs don’t match any of the provider’s socket names.
# See the consumer's plugs:shc inspect app <app> # lists plugs + their target socket names
# See the provider's sockets:shc inspect stack <provider-stack>If the names don’t match (e.g. your plug wants postgres but the
provider exports database), use explicit syntax:
shc install myapp -i postgres=provider-stack.databaseSymptom: X404007PLGNOF / X404006SOKNOF
Section titled “Symptom: X404007PLGNOF / X404006SOKNOF”Plug or socket name in your -i spec doesn’t exist on the named
stack. Typo or stale spec. Verify names with shc inspect stack.
Symptom: Consumer renders a literal {{ integrations.X.Y }}
Section titled “Symptom: Consumer renders a literal {{ integrations.X.Y }}”The integration didn’t register before compose rendered, so the template variable was unset. This almost always means the integration phase job failed silently.
shc integration ls <stack> # verify the edge existsshc job history # find the failed integration-phase jobshc job show <run-id> # get full stdout/stderrRe-run the integration:
shc update <stack> -i <plug>! # detach the broken edgeshc update <stack> -i <plug>=<provider> # re-wire itSymptom: X400954XNDSCK (cross-node socket blocked)
Section titled “Symptom: X400954XNDSCK (cross-node socket blocked)”The provider’s socket has local_only: true, meaning it can only
serve consumers on the same node. Your consumer is on a different
node.
Options:
- Install the consumer on the same node
(
shc install <app> --where 'node=<same-as-provider>') - Remove the
local_onlyrestriction in the provider app’s socket YAML (only appropriate if the exposed service is actually mesh-reachable)
Backup and restore issues
Section titled “Backup and restore issues”Symptom: Backup fails with X500908HKFAIL
Section titled “Symptom: Backup fails with X500908HKFAIL”A pre_backup or post_backup hook (or a backup converter)
failed. SHC logs the job error but does not abort the backup,
so this is a warning, not a fatal error, unless the backup
evidently lacks expected content.
shc job history # find the failed hook runshc job show <run-id> # full outputThen:
- Run the job manually against the same stack to reproduce
- Check if the job expects a service that’s not yet running (run it at a different hook phase; see lifecycle-hooks: allowed contexts per hook)
Symptom: “restic repo not reachable”
Section titled “Symptom: “restic repo not reachable””Backup failed before capture because restic couldn’t access its repo.
- Confirm
backup.repoin your config points at a reachable location, and thatbackup.passwordis set; they are required together, and either one alone is refused - If
backup.passwordholds aref+vault://reference, confirm it resolves: an unresolvable reference (locked vault, absent key) refuses rather than falling back. Unlock withshc vault unlock, then read the path back withshc vault get backup/password - For NFS/local shared repos, confirm the mount is present on every node (the node doing the backup is the node the stack lives on). See storage-providers: NFS.
Symptom: Restore succeeds but stack won’t start
Section titled “Symptom: Restore succeeds but stack won’t start”Restore completed (metadata, volumes, vault secrets, integrations) but the restored stack’s containers fail to come up.
Most common causes:
- Integrations didn’t replay. Restore re-creates integration
rows but may skip re-running the
integrationphase jobs if the provider is unavailable at restore time. Checkshc integration ls <stack>: if edges exist butwait_forjobs aren’t present in provider DBs, re-issue the edge so the integration phase runs again:Terminal window shc update <stack> -i <plug>=<provider> - Vault scope mismatch. Restoring into a different tenant or
environment means
ref+vault://...?scope=deploymentreferences resolve to nothing until you set values in the new scope. See vault: scopes. - Image version drift. The app was installed at v1 but now
resolves to v2, and the restored data is v1-compatible. Pin
versions in
app.yamlor use a repo at a fixed ref.
Cluster issues
Section titled “Cluster issues”Symptom: shc node join hangs at “enrolling”
Section titled “Symptom: shc node join hangs at “enrolling””The joiner needs HTTPS 443 to the cluster’s public enrolment route through Traefik, then UDP 4244 between node underlay addresses for mesh0.
# On the joining node, check TCP connectivity to the HTTPS entrypoint:nc -zv <cluster-host> 443
# On the leader:ss -tlnp | grep ':443' # HTTPS entrypointss -ulnp | grep ':4244' # mesh0 UDP listenershc cluster mesh statusshc cluster mesh diagnose # peer internal APIs over mesh0A listening UDP socket alone does not prove reachability. Inspect mesh handshake status and diagnose output after enrolment.
See cluster join flow for the full handshake.
Symptom: shc node ls shows a node as unhealthy forever
Section titled “Symptom: shc node ls shows a node as unhealthy forever”Either the node is actually dead (see disaster recovery: Scenario 3) or the healthcheck probe can’t reach it.
# On any healthy voter:shc cluster status # node counts, voter/quorum state, leadershc cluster ping <node> # probe that node directlyshc cluster mesh diagnose # this node probes peer internal APIs over mesh0
# Run this check from the "unhealthy" node itself if you have shell:shc node ls # if it also sees itself as lonely, # mesh is broken from its sideSymptom: Writes return 503 / X500021CLSINT
Section titled “Symptom: Writes return 503 / X500021CLSINT”Raft lost quorum. Recover per disaster recovery: Scenario 2.
Symptom: Certificate rotation failing
Section titled “Symptom: Certificate rotation failing”Logs show cert expiry warnings or rotation errors.
shc cluster cert ls # report cert ages across the clustershc cluster cert rotate # force-rotate everything that's dueshc cluster cert rotate --type mtls # rotate only one categorySee disaster recovery: Scenario 5.
Vault issues
Section titled “Vault issues”Symptom: an operation fails with “vault is locked”
Section titled “Symptom: an operation fails with “vault is locked””X503903VLTPWM. This reports a runtime lock state, not a
configuration fault: the vault exists and is initialized, but no
master key is held in memory right now. Nothing is missing from your
config.
On a cluster follower this is usually normal and self-healing.
Followers deliberately hold no copy of the master password. They
boot locked and pull the unlock from an already-unlocked peer within
seconds. A burst of X503903VLTPWM in the first moments after
systemctl restart shc is the expected shape; the pull loop logs
vault unlock pull: loop armed → knocking peers → peer accepted push-back request, then goes quiet. Silence is the success signal.
A command typed on the follower in those seconds waits for the unlock,
up to 20 s, instead of being refused X503906VLTLCK. The refusal comes
at once only on a node no peer can unlock: a single node, or one whose
peers are all locked.
Confirm rather than assume: shc vault status prints a two-row table
whose header contains the word LOCKED, so a naive grep reports a
false positive:
shc vault statusIf a node stays locked past a minute:
-
Follower: it has no unlocked peer to pull from, or the peer transport is down. Check that another node reports
locked:false, then look for the pull-loop lines above. Do not “fix” this by puttingSHC_VAULT_PASSWORDon the follower, keeping the master password off every non-genesis node is deliberate. -
Genesis / single node: this one does carry the password, because at genesis there is no unlocked peer to pull from. It is sealed with
systemd-credsin/etc/systemd/system/shc.service.d/vault-autounlock.cred, which shcd opens at boot. On launchd it is a System keychain item (io.selfhostedcloud.shc.vault-unlock). If the credential is absent, or shcd’s log says it did not open (the host key or the TPM it was sealed to changed, or the keychain item could not be read), unlock manually. The unlock also seals a fresh credential, so the next restart auto-unlocks (see vault: the sealed auto-unlock credential). Thevault.autounlock_at_restrow ofshc doctor run -c vaultsays which of these the node is in. On a node whose systemd is older than 250 there is no credential at all (X200052VLTSKP), and every restart needs the manual unlock. If shcd was started withSHC_VAULT_PASSWORDset it refuses withX400975VLTRFEand does not fall back; move the node with the steps in that section. The rewrite of the secret file is the operator’s. Let the command prompt (echo is suppressed on a terminal), or pipe the password in. Never pass it as an argument, which leaks it into shell history andps:Terminal window sudo shc vault unlock # prompts, no echocat /path/to/password-file | sudo shc vault unlock-p/--passwordtakes the password value, not a path to one:-p /path/to/filetries to unlock with the literal string/path/to/fileand fails. There is no file form of the flag.
See vault: unlocking.
Symptom: X500961VLTCRP (vault crypto failure)
Section titled “Symptom: X500961VLTCRP (vault crypto failure)”The stored blob can’t be decrypted with the current master password.
The {op} token names the step that gave up (base64 decode,
ciphertext too short, bad fernet key), so quote it when reporting.
A value that is not in the envelope format at all fails earlier and
differently, with X400980VLTFMT.
- You recently rotated and the rotation didn’t finish → restart daemon, retry
- You restored a backup from a different cluster → rotate to the backup’s original password, then rotate again to your new one
- Hardware corruption of the state database (
/var/lib/shc/shc.db, or/var/lib/shc/rqlitein cluster mode) → restore from a clean backup
Symptom: ref+vault://key renders unresolved
Section titled “Symptom: ref+vault://key renders unresolved”The resolver returned None for the key. Checks:
- Does the value exist?
shc vault exists <key>at the exact(scope, tenant, stack, environment)you expect. - Is the scope in the URI what you think?
?scope=tenantvs?scope=deploymentresolve against different keys. - Is the vault healthy?
shc doctor run --category vaultreports lock state and flags broken (empty-value) rows. An unreachable external provider surfaces asX503902VLTNAV(vault not available) in the daemon log. - Did you autogen?
?autogenonly generates on first render; if the first render failed (vault write error), the key may still be absent on retry. Check forX500901OPFAILin logs.
Daemon won’t start
Section titled “Daemon won’t start”Check order
Section titled “Check order”# 1. Is there actually a process?systemctl status shc
# 2. What did it log?journalctl -u shc -n 200
# 3. Can it reach its state dir?ls -la /var/lib/shc/sudo -u shc test -w /var/lib/shc && echo OK
# 4. Are required ports free?ss -tlnp | grep -E '4001|4002|4003'ss -ulnp | grep 4244
# 5. Is the raft engine's data corrupt? (cluster mode)ls -la /var/lib/shc/rqlite/# Look for a fresh 'raft.db' and old logs. If this is a cluster# member whose peers are gone, see disaster-recovery Scenario 2.Common startup failures
Section titled “Common startup failures”- “address already in use”: another process owns the port.
Find it with
ss -tlnp. SHC’s internal API is TCP 4003 by default; change viainternal_api.portin config. - “permission denied” on
/var/lib/shc/...: systemd user/group mismatch. Verify withstat /var/lib/shcandid shc. The state dir must be owned by theshcuser. - “vault is locked” (
X503903VLTPWM): a runtime lock state, not a missing setting; on a follower it is normally transient. See vault section above. - “raft: failed to open snapshot”: the raft engine’s snapshot is corrupt. See disaster recovery: Option C, restore from backup.
Getting more signal
Section titled “Getting more signal”If none of the above helps, turn up the logging and re-run:
# Run the daemon in foreground with debug logging:systemctl stop shcsudo -u shc SHC_LOG_LEVEL=debug shcd…and inspect the full structured log (not just human messages)
via shc logs --follow (on a separate shell) while reproducing the
issue.
For production incidents, gather the diagnostic surfaces into one place before opening an issue:
shc doctor run > doctor.txt # every registered diagnosisshc version > version.txt # version + build infoshc cluster status > cluster.txt # voter/quorum state, leadershc node ls > nodes.txt # membership + node healthshc events --audit -n 200 > audit.log # recent audited actionsjournalctl -u shc --since -2h > daemon.log # structured daemon logRelated
Section titled “Related”- Error reference: every error code SHC emits
- Disaster recovery: cluster-level failures
- Observability: logs, metrics, traces
- Flow docs: step-by-step pipeline traces