Doctor
shc status tells you that something is wrong. shc doctor tells you what.
It runs a list of diagnoses on the daemon, each with a name, a category and,
for some, a prescription. Start with the list.
$ shc doctor lsNAME CATEGORY FIX DESCRIPTION...app.older-renders app - name each older render this node holds beside the deployment's committed renderapp.telemetry-export app - report a managed telemetry export held back by an unreadable system vault...backup.freshness backup - report schedules whose newest archive is older than their interval...database.orphaned-dirs database - check for deployment directories with no db record...deployments.missing-compose deployments - check for stacks with template but no rendered compose file...state.oom-kills state - check this node's kernel log and recorded container exits for processes the kernel OOM killer has reaped...system.docker-daemon system - check docker daemon is running and accessible...FIX is the class of each diagnosis’s prescription. benign and
mutating prescriptions run when you pass --fix. A destructive one
never runs from --fix; you authorise it by naming its diagnosis. A -
means the diagnosis only reports.
Run everything. shc doctor run with no category runs every diagnosis and
exits non-zero while any of them fails. A cluster that was just built fails
some by its own state: nothing has been backed up and no schedule is armed,
the backup scratch directory still sits under the daemon’s state home, these
machines resolve names without systemd-resolved, and automatic volume
failover is off until it is switched on. Each of those is a finding with
the command that clears it under it, so the whole run is read here, not
typed.
$ shc doctor run-c runs one category. The system category checks the daemon’s own
directories, the docker daemon and the control plane’s voters, and a fresh
cluster passes all of it.
$ shc doctor run -c system...The database category checks the daemon’s own records of its deployments.
$ shc doctor run -c database[pass] database.orphaned-dirs[pass] database.orphaned-stacks[pass] database.orphaned-vdeployments[pass] database.parked-stacks[pass] database.stale-status[pass] database.storage[pass] deployment.orphaned-coordinates✔ ran 7 diagnoses, 7 passedNow break something. The daemon keeps each deployment’s files under its state directory, one directory per tenant, deployment and environment. Make a directory there that no deployment owns.
$ mkdir -p /var/lib/shc/deployments/prod/ghost/prod$ shc doctor run -c database[warn] database.orphaned-dirs warning: Deployment directory exists on node node-15826b16f572 but no DB record: prod/ghost/prod /var/lib/shc/deployments/prod/ghost/prod[pass] database.orphaned-stacks[pass] database.orphaned-vdeployments[pass] database.parked-stacks[pass] database.stale-status[pass] database.storage[pass] deployment.orphaned-coordinateswarning: ran 7 diagnoses, 6 passed, 1 warningA finding sits under its diagnosis with its severity, its message and the
detail that names the thing. The tag in front of a diagnosis is its worst
finding: [info], [warn] or [fail]. A warning exits 3 and a failure 1.
This diagnosis has no prescription: deleting a directory under a tenant is
yours to do, not doctor’s.
$ rm -r /var/lib/shc/deployments/prod/ghost$ shc doctor run -c database[pass] database.orphaned-dirs[pass] database.orphaned-stacks[pass] database.orphaned-vdeployments[pass] database.parked-stacks[pass] database.stale-status[pass] database.storage[pass] deployment.orphaned-coordinates✔ ran 7 diagnoses, 7 passedSome diagnoses carry a prescription. state.container-mismatch is one: it
fails when the database says a deployment runs and docker holds no container
for it, and its fix restates the row as degraded. Try to make it fail.
placed is the notes app, one postgres container, and the rule page 04 of
the app ladder gave it keeps it in learn/home, on home01.
Remove the container behind SHC’s back.
$ docker ps --filter label=shc.stack=placed --format '{{.Names}}'prod_prod_placed-postgres-r0-postgres-r0-1$ docker rm -f prod_prod_placed-postgres-r0-postgres-r0-1prod_prod_placed-postgres-r0-postgres-r0-1The daemon on home01 reads docker’s own event as the container goes and
marks placed degraded itself, within seconds. The row and docker never
disagree, so the diagnosis has nothing to find and passes.
$ shc ls -s placedSTACK TENANT ENV APP STATUS NODEplaced prod prod notes degraded home01warning: 1 of 1 deployments are not running: 1 degraded$ shc doctor run -c state[pass] state.container-mismatch[pass] state.oom-kills[pass] state.unrecorded-containers✔ ran 3 diagnoses, 3 passedThe health reconciler notices a degraded deployment and starts its
containers again, usually within a minute or two. Run shc ls again until
placed reads running.
$ shc ls -s placedSTACK TENANT ENV APP STATUS NODEplaced prod prod notes running home01--fix applies the benign and mutating prescriptions of the diagnoses
that fail and says under each finding what it did. Nothing fails here, so it
changes nothing.
$ shc doctor run -c state --fix...[pass] state.container-mismatch[pass] state.oom-kills[pass] state.unrecorded-containers✔ ran 3 diagnoses, 3 passedA fix runs as an operation on the daemon. Before the results the command
names it, and shc operation ls lists it while it runs. Interrupting the
command does not stop the fix: shc doctor follow with that id waits for it
again and prints the same results.
$ shc doctor follow 3f2b6c1e-8d4a-4f0e-9b7c-2a5d1e6f8c90When a diagnosis with a prescription does fail, the run prints the fix under
the finding, with its class and whether it can be undone, and --fix
applies it. system.directories is the one to read first: it fails when one
of the daemon’s own directories is missing, and its benign prescription
creates the directory again. Doctor checks the node it is typed on, so break
this one on obs01. The daemon’s log directory is empty there, because these
machines log to the journal. Remove it and run the category.
$ rmdir /var/log/shc$ shc doctor run -c system[pass] system.cluster-quorum[fail] system.directories error: Directory missing: /var/log/shc logs_dir fix available [benign, reversible]: systemd-tmpfiles makes each missing state directory shc's tmpfiles.d declares, root:shc 0750, on the host; mkdir makes any other 0750: creates only, removes nothing fix not requested, pass --fix[pass] system.docker-address-pools[pass] system.docker-daemon[pass] system.leader-workwarning: ran 5 diagnoses, 4 passed, 1 failedThe finding carries its fix, the fix’s class and what it does. Apply it. The
daemon’s unit cannot write under /var/log, so the fix has systemd make the
directory with the owner and mode the package declares, and the finding says
the daemon needs a restart before it can write there.
$ shc doctor run -c system --fix...[pass] system.cluster-quorum[fixed] system.directories error+fixed: Directory missing: /var/log/shc shcd cannot write it until shc restarts: systemctl restart shc[pass] system.docker-address-pools[pass] system.docker-daemon[pass] system.leader-work✔ ran 5 diagnoses, 4 passed, 1 fixed$ stat -c '%A %U %G %n' /var/log/shcdrwxr-x--- root shc /var/log/shc$ systemctl restart shcobs01 is not a voter, so restarting its daemon moves no leader and stops no container. Run the category again until the daemon answers.
$ shc doctor run -c system[pass] system.cluster-quorum[pass] system.directories[pass] system.docker-address-pools[pass] system.docker-daemon[pass] system.leader-work✔ ran 5 diagnoses, 5 passedA destructive prescription is named, never implied. database.orphaned-stacks
would delete a deployment row that has no directory. Authorise it by name, and
-y skips the confirmation it would otherwise ask for. Nothing here is
orphaned, so it passes and deletes nothing.
$ shc doctor run -c database --fix-destructive database.orphaned-stacks -ywarning: database.orphaned-stacks applies a destructive fix: DELETE FROM deployment WHERE id = ?, one row per flagged deployment, permanently removing the control plane's record of it, recoverable-from-backup, and the estate forgets the deployment entirely: it disappears from `shc ls`, its backups stop being discoverable by name, and its config/secret rows lose the row they hang off. Nothing on disk is removed, so a wrongly-deleted row leaves data no verb can re-adopt: only a control-plane database restore puts it back. The finding it acts on is an os.Stat miss, which a missing mount produces just as readily as a genuine orphan....[pass] database.orphaned-dirs[pass] database.orphaned-stacks[pass] database.orphaned-vdeployments[pass] database.parked-stacks[pass] database.stale-status[pass] database.storage[pass] deployment.orphaned-coordinates✔ ran 7 diagnoses, 7 passedOne diagnosis stands apart from the sweep. shc doctor origin-probe fetches
a page of a running deployment under several Host headers and sorts every
absolute origin it finds into derived, pinned in the render, pinned in the
app’s own state, or foreign. Run it before you move a deployment to a new
host. web serves a static page with no absolute link in it, so the probe
has nothing to classify there and says so instead of passing it. cloud
answers every host with a page that links to Nextcloud’s own sites, so probe
that one. It dials the service inside the deployment’s network, so name the
service, and give it two hosts to compare. An origin that is the same under
both and is not one of the deployment’s own names is foreign: reported, not
a fault. The probe exits non-zero only for a seed-once origin, one the app
persisted at first run.
$ shc doctor origin-probe cloud --service nextcloud --path / --host-header one.example --host-header two.exampleORIGIN VERDICT WHYprod.nextcloud.cloud seed-once constant across every probe host and equal to this stack's in-cluster network alias, so it was captured at first run rather than configurednextcloud.com foreign constant across every probe host but not an identity of this deploymentwww.enable-javascript.com foreign constant across every probe host but not an identity of this deploymentThe probe takes the tenant and environment flags every verb takes, -t
and -e; prod is this cluster’s default for both, so naming the tenant
answers the same.
$ shc doctor origin-probe cloud -t prod --service nextcloud --path / --host-header one.example --host-header two.exampleORIGIN VERDICT WHYprod.nextcloud.cloud seed-once constant across every probe host and equal to this stack's in-cluster network alias, so it was captured at first run rather than configurednextcloud.com foreign constant across every probe host but not an identity of this deploymentwww.enable-javascript.com foreign constant across every probe host but not an identity of this deployment--edge probes through this node’s ingress over https instead of inside
the deployment’s network, for an app that redirects plain http, and only
for a host the ingress routes. This cluster was initialised without
--ingress-host, so no deployment has one and the edge has nothing to
route.
$ shc doctor origin-probe cloud --edge --path /This page leaves every deployment as it found it: placed runs again on
home01, the ghost directory is gone, and obs01 has its log directory back.
Next: Events.