Skip to content
SHC Docs

Doctor

shc status tells you that something is wrong. shc doctor tells you what. It runs a list of diagnoses on the daemon, each with a name, a category and, for some, a prescription. Start with the list.

home01
$ shc doctor ls
NAME CATEGORY FIX DESCRIPTION
...
app.older-renders app - name each older render this node holds beside the deployment's committed render
app.telemetry-export app - report a managed telemetry export held back by an unreadable system vault
...
backup.freshness backup - report schedules whose newest archive is older than their interval
...
database.orphaned-dirs database - check for deployment directories with no db record
...
deployments.missing-compose deployments - check for stacks with template but no rendered compose file
...
state.oom-kills state - check this node's kernel log and recorded container exits for processes the kernel OOM killer has reaped
...
system.docker-daemon system - check docker daemon is running and accessible
...

FIX is the class of each diagnosis’s prescription. benign and mutating prescriptions run when you pass --fix. A destructive one never runs from --fix; you authorise it by naming its diagnosis. A - means the diagnosis only reports.

Run everything. shc doctor run with no category runs every diagnosis and exits non-zero while any of them fails. A cluster that was just built fails some by its own state: nothing has been backed up and no schedule is armed, the backup scratch directory still sits under the daemon’s state home, these machines resolve names without systemd-resolved, and automatic volume failover is off until it is switched on. Each of those is a finding with the command that clears it under it, so the whole run is read here, not typed.

explained, not walked: a fresh learn cluster fails the backup, dns and failover diagnoses it is built to fail (SHC-2169), so the whole run exits non-zero
$ shc doctor run

-c runs one category. The system category checks the daemon’s own directories, the docker daemon and the control plane’s voters, and a fresh cluster passes all of it.

home01
$ shc doctor run -c system
...

The database category checks the daemon’s own records of its deployments.

home01
$ shc doctor run -c database
[pass] database.orphaned-dirs
[pass] database.orphaned-stacks
[pass] database.orphaned-vdeployments
[pass] database.parked-stacks
[pass] database.stale-status
[pass] database.storage
[pass] deployment.orphaned-coordinates
✔ ran 7 diagnoses, 7 passed

Now break something. The daemon keeps each deployment’s files under its state directory, one directory per tenant, deployment and environment. Make a directory there that no deployment owns.

home01
$ mkdir -p /var/lib/shc/deployments/prod/ghost/prod
$ shc doctor run -c database
[warn] database.orphaned-dirs
warning: Deployment directory exists on node node-15826b16f572 but no DB record: prod/ghost/prod
/var/lib/shc/deployments/prod/ghost/prod
[pass] database.orphaned-stacks
[pass] database.orphaned-vdeployments
[pass] database.parked-stacks
[pass] database.stale-status
[pass] database.storage
[pass] deployment.orphaned-coordinates
warning: ran 7 diagnoses, 6 passed, 1 warning

A finding sits under its diagnosis with its severity, its message and the detail that names the thing. The tag in front of a diagnosis is its worst finding: [info], [warn] or [fail]. A warning exits 3 and a failure 1. This diagnosis has no prescription: deleting a directory under a tenant is yours to do, not doctor’s.

home01
$ rm -r /var/lib/shc/deployments/prod/ghost
$ shc doctor run -c database
[pass] database.orphaned-dirs
[pass] database.orphaned-stacks
[pass] database.orphaned-vdeployments
[pass] database.parked-stacks
[pass] database.stale-status
[pass] database.storage
[pass] deployment.orphaned-coordinates
✔ ran 7 diagnoses, 7 passed

Some diagnoses carry a prescription. state.container-mismatch is one: it fails when the database says a deployment runs and docker holds no container for it, and its fix restates the row as degraded. Try to make it fail. placed is the notes app, one postgres container, and the rule page 04 of the app ladder gave it keeps it in learn/home, on home01. Remove the container behind SHC’s back.

home01
$ docker ps --filter label=shc.stack=placed --format '{{.Names}}'
prod_prod_placed-postgres-r0-postgres-r0-1
$ docker rm -f prod_prod_placed-postgres-r0-postgres-r0-1
prod_prod_placed-postgres-r0-postgres-r0-1

The daemon on home01 reads docker’s own event as the container goes and marks placed degraded itself, within seconds. The row and docker never disagree, so the diagnosis has nothing to find and passes.

home01
$ shc ls -s placed
STACK TENANT ENV APP STATUS NODE
placed prod prod notes degraded home01
warning: 1 of 1 deployments are not running: 1 degraded
$ shc doctor run -c state
[pass] state.container-mismatch
[pass] state.oom-kills
[pass] state.unrecorded-containers
✔ ran 3 diagnoses, 3 passed

The health reconciler notices a degraded deployment and starts its containers again, usually within a minute or two. Run shc ls again until placed reads running.

home01
$ shc ls -s placed
STACK TENANT ENV APP STATUS NODE
placed prod prod notes running home01

--fix applies the benign and mutating prescriptions of the diagnoses that fail and says under each finding what it did. Nothing fails here, so it changes nothing.

home01
$ shc doctor run -c state --fix
...
[pass] state.container-mismatch
[pass] state.oom-kills
[pass] state.unrecorded-containers
✔ ran 3 diagnoses, 3 passed

A fix runs as an operation on the daemon. Before the results the command names it, and shc operation ls lists it while it runs. Interrupting the command does not stop the fix: shc doctor follow with that id waits for it again and prints the same results.

explained, not walked: needs a fix still running after its command was interrupted
$ shc doctor follow 3f2b6c1e-8d4a-4f0e-9b7c-2a5d1e6f8c90

When a diagnosis with a prescription does fail, the run prints the fix under the finding, with its class and whether it can be undone, and --fix applies it. system.directories is the one to read first: it fails when one of the daemon’s own directories is missing, and its benign prescription creates the directory again. Doctor checks the node it is typed on, so break this one on obs01. The daemon’s log directory is empty there, because these machines log to the journal. Remove it and run the category.

obs01
$ rmdir /var/log/shc
$ shc doctor run -c system
[pass] system.cluster-quorum
[fail] system.directories
error: Directory missing: /var/log/shc
logs_dir
fix available [benign, reversible]: systemd-tmpfiles makes each missing state directory shc's tmpfiles.d declares, root:shc 0750, on the host; mkdir makes any other 0750: creates only, removes nothing
fix not requested, pass --fix
[pass] system.docker-address-pools
[pass] system.docker-daemon
[pass] system.leader-work
warning: ran 5 diagnoses, 4 passed, 1 failed

The finding carries its fix, the fix’s class and what it does. Apply it. The daemon’s unit cannot write under /var/log, so the fix has systemd make the directory with the owner and mode the package declares, and the finding says the daemon needs a restart before it can write there.

obs01
$ shc doctor run -c system --fix
...
[pass] system.cluster-quorum
[fixed] system.directories
error+fixed: Directory missing: /var/log/shc
shcd cannot write it until shc restarts: systemctl restart shc
[pass] system.docker-address-pools
[pass] system.docker-daemon
[pass] system.leader-work
✔ ran 5 diagnoses, 4 passed, 1 fixed
$ stat -c '%A %U %G %n' /var/log/shc
drwxr-x--- root shc /var/log/shc
$ systemctl restart shc

obs01 is not a voter, so restarting its daemon moves no leader and stops no container. Run the category again until the daemon answers.

obs01
$ shc doctor run -c system
[pass] system.cluster-quorum
[pass] system.directories
[pass] system.docker-address-pools
[pass] system.docker-daemon
[pass] system.leader-work
✔ ran 5 diagnoses, 5 passed

A destructive prescription is named, never implied. database.orphaned-stacks would delete a deployment row that has no directory. Authorise it by name, and -y skips the confirmation it would otherwise ask for. Nothing here is orphaned, so it passes and deletes nothing.

home01
$ shc doctor run -c database --fix-destructive database.orphaned-stacks -y
warning: database.orphaned-stacks applies a destructive fix: DELETE FROM deployment WHERE id = ?, one row per flagged deployment, permanently removing the control plane's record of it, recoverable-from-backup, and the estate forgets the deployment entirely: it disappears from `shc ls`, its backups stop being discoverable by name, and its config/secret rows lose the row they hang off. Nothing on disk is removed, so a wrongly-deleted row leaves data no verb can re-adopt: only a control-plane database restore puts it back. The finding it acts on is an os.Stat miss, which a missing mount produces just as readily as a genuine orphan.
...
[pass] database.orphaned-dirs
[pass] database.orphaned-stacks
[pass] database.orphaned-vdeployments
[pass] database.parked-stacks
[pass] database.stale-status
[pass] database.storage
[pass] deployment.orphaned-coordinates
✔ ran 7 diagnoses, 7 passed

One diagnosis stands apart from the sweep. shc doctor origin-probe fetches a page of a running deployment under several Host headers and sorts every absolute origin it finds into derived, pinned in the render, pinned in the app’s own state, or foreign. Run it before you move a deployment to a new host. web serves a static page with no absolute link in it, so the probe has nothing to classify there and says so instead of passing it. cloud answers every host with a page that links to Nextcloud’s own sites, so probe that one. It dials the service inside the deployment’s network, so name the service, and give it two hosts to compare. An origin that is the same under both and is not one of the deployment’s own names is foreign: reported, not a fault. The probe exits non-zero only for a seed-once origin, one the app persisted at first run.

home01
$ shc doctor origin-probe cloud --service nextcloud --path / --host-header one.example --host-header two.example
ORIGIN VERDICT WHY
prod.nextcloud.cloud seed-once constant across every probe host and equal to this stack's in-cluster network alias, so it was captured at first run rather than configured
nextcloud.com foreign constant across every probe host but not an identity of this deployment
www.enable-javascript.com foreign constant across every probe host but not an identity of this deployment

The probe takes the tenant and environment flags every verb takes, -t and -e; prod is this cluster’s default for both, so naming the tenant answers the same.

home01
$ shc doctor origin-probe cloud -t prod --service nextcloud --path / --host-header one.example --host-header two.example
ORIGIN VERDICT WHY
prod.nextcloud.cloud seed-once constant across every probe host and equal to this stack's in-cluster network alias, so it was captured at first run rather than configured
nextcloud.com foreign constant across every probe host but not an identity of this deployment
www.enable-javascript.com foreign constant across every probe host but not an identity of this deployment

--edge probes through this node’s ingress over https instead of inside the deployment’s network, for an app that redirects plain http, and only for a host the ingress routes. This cluster was initialised without --ingress-host, so no deployment has one and the edge has nothing to route.

explained, not walked: --edge needs a host the ingress routes, and a cluster initialised without --ingress-host has none
$ shc doctor origin-probe cloud --edge --path /

This page leaves every deployment as it found it: placed runs again on home01, the ghost directory is gone, and obs01 has its log directory back.

Next: Events.