Skip to content
SHC Docs

Doctor module

Layer 4. Primitives. Primitives Checks have no owned record.

The doctor module is a diagnose-and-prescribe framework. It runs a registered list of checks, each of which can optionally carry a repair. The CLI entry point is shc doctor run: inspection-only by default; repairs run only with an explicit --fix.

Unlike runtime health (shc health, shc hud), doctor is intended for install-time, post-recovery, and offline cleanup, the kind of issues that accumulate when containers/volumes get cleaned up out-of-band or when a crashed install leaves orphan state behind.

Source: modules/doctor/: framework.go holds the types and the verdict tree, register.go seeds the core diagnoses from diagnoses/, bridge.go adopts the probes sibling modules publish through the central registry, and router.go exposes the HTTP endpoints.

  1. Framework types: Diagnosis, Finding, DiagnosisResult, plus the Severity and Verdict enums that callers render.
  2. Runner: Runner iterates the registered diagnoses, captures panics/errors into SKIPPED results, and applies prescribe callbacks only when fix is requested.
  3. Registration: the doctor package’s own init() seeds the core diagnoses; sibling modules (vault / storage / repo) publish slim (status, hint, err) probes via registry.Diagnoses / registry.DiagnosesWithDeps, and the daemon adopts those at wire time via doctor.RegisterModuleDiagnoses (bridge.go).
  4. Built-in diagnoses: ten checks across five categories covering Docker/Filesystem, DB-backed state, compose-rendered deployment dirs, DB↔container drift, and keycloak realms.

Every Diagnosis is a record of:

type Diagnosis struct {
Name string
Description string
Category string
Diagnose DiagnoseFunc
Prescribe Prescription // nil means inspection only
}

DiagnoseFunc is func(ctx context.Context) ([]Finding, error). Zero findings is a pass. Prescribe is a remedy.Prescription. A non-nil value carries a class: remedy.Benign, remedy.Mutating, or remedy.Destructive. The runner sets Finding.Fixed when a repair applied.

The runner grades a result by its worst finding (Grade): info reads INFO, warning reads WARN, error and critical read FAIL, and no finding reads PASS. Under --fix (or {"fix": true} on the API) the findings a prescription left unfixed decide the grade, and a result whose every finding worse than info came back Fixed reads FIXED. A diagnose error reads SKIPPED, and X200985DCRRTM reads INAPPLICABLE. The default run is the preview: results whose skipped prescription could repair them carry fixable=true. The ladder, what each verdict prints and the exit status it gives are in reference/cli/doctor.

DoctorRunner.Run isolates each diagnosis. A panic or an error from Diagnose becomes one error-severity finding and a SKIPPED verdict for that check. The rest of the sweep still runs. X200985DCRRTM is the one error that is not a skip: the check reads docker on a node whose runtime runs none, so its verdict is INAPPLICABLE with no finding, and the condition sweep counts it as observed. A prescribe error is appended as an error-severity finding, so that check reads FAIL.

Registered from modules/doctor/diagnoses/:

CategoryDiagnosis namePrescribe?What it checks
systemsystem.docker-daemonnoDocker daemon reachable
systemsystem.docker-address-poolsyesdefault-address-pools is provisioned and has headroom (never restarts dockerd)
systemsystem.directoriesyesRequired SHC directories exist and are writable
systemsystem.gpunoAn NVIDIA GPU the driver shows that the node’s runtime cannot pass to a container, and what the node lacks (the container toolkit, a CDI spec)
systemsystem.auth-sectionnoThis node’s clusterfs auth section is fresh: a proof older than five minutes, or of another digest than it serves, makes the node’s forward-auth answer every gated request 503 stale_replica, and the finding names the node
systemsystem.clusterfs-snapshotnoThe store snapshot each clusterfs build copies into memory on this node stays under 64 MiB: past it every build on every node pays that copy, and the finding names the node, the rows and the size
databasedatabase.orphaned-stacksyesDB stack rows whose deployment dir is gone
databasedatabase.orphaned-dirsnoDeployment dirs with no matching DB row
databasedatabase.stale-statusyesstatus / desired words disagree with running state
databasedatabase.storagenoDatabase file present, sized sanely, writable
statestate.container-mismatchyesDB says running but no container (or vice versa)
statestate.oom-killsnoContainers on this node the kernel OOM killer reaped. The kernel log names every kill at a container’s own memory limit in the last 24 hours, worker kills that leave the container running and incus crash loops included, with the tenant, deployment, service, limit, peak and count. An OOM exit the node recorded is reported when the log no longer holds it, and is the only record of a container with no limit of its own that the host ran out of memory under. Two or more kills fail
statestate.unrecorded-containersnoSHC-labelled containers on this node with no deployment row or no service row, split into debris (docker rm), rowless declared services (shc update) and undeclared containers
deploymentsdeployments.missing-composeyesStack has a template but no rendered compose.yaml
deploymentsdeployments.unhealthynoA deployment wanted running reads degraded, failed or stopped for a sweep (deployment.Unhealthy)
deploymentsdeployments.scheduled-jobsnoA deployment wanted running whose scheduled standalone job failed its last finished run, named with the job, the run, when and its error; the next passing run clears it (deployment.JobFailed)
deploymentsdeployments.replicas-downnoA replica wanted running stood assigned, starting, unready or failed on its node for 10 minutes, named with its node, state and reason
deploymentsdeployments.replicas-flappingnoA replica whose readiness flipped 4 times in 10 minutes, named with its node and whether routing holds it out (deployment.ReplicaFlapping)
deploymentsdeployments.from-trustnoA deployment running the app a --from put in whose sidecar does not record who chose that source, so it renders with its own source’s trust; the finding gives the shc update --from that records it
keycloakkeycloak.realmsyesEvery tenant has a keycloak realm
doctordoctor.selfnoThe doctor’s own dependency seams are wired

On the daemon the table additionally carries the module probes adopted from the central registry (vault.status, vault.secrets, vault.crypto_format, storage.local, repo.sync-state): inspection-only entries whose category is the publishing module’s name.

The categories are free-form strings; the CLI’s --category/-c flag filters by exact match.

Every check states its scope in code (Scope on the diagnosis): cluster when its answer is the same from every node, node when it reads the node that runs it (disk, container engine, network, local services). shc doctor ls prints it in its SCOPE column.

shc doctor run on a cluster reports the cluster. The node it reaches runs each cluster check once and its own node checks, and asks every other roster node at once over the internal RPC doctor.node_run for its node checks in the run’s category. A node check prints one row per node, the node’s name in the last column:

Terminal window
$ shc doctor run -c dns
[fail] dns.host-routing incus01
error: no route to 10.236.0.1
[pass] dns.host-routing incus02
[skipped] doctor.node-checks incus03
error: incus03 did not answer: connection refused

A node the roster reads unreachable is not dialled, and it and a node that does not answer within 90 seconds each print one skipped doctor.node-checks row saying why. When the roster cannot be read, one doctor.node-checks row says so and the node checks answer for the node reached alone. The other nodes answer diagnose-only: a node applies its own fixes, so another node’s row carries its fix withheld with the command to run there. The exit code and the summary count every row on every node.

The condition sweep runs on the leader. It runs the cluster checks once, runs the node checks for itself, and asks every other node for its node checks over the internal RPC doctor.node_checks. A node-scoped condition’s subject carries the node id, so each node holds its own condition and its own “since”. A node that does not answer, or that the roster reads unreachable, reads check.Unobservable for each node check and its open node conditions turn Unknown; none of them clears until the node answers again. A node that leaves the roster takes its node conditions with it.

  • api.public-listener
  • app.appyaml-drift
  • app.backup-v0121-meaning
  • app.cert-homes-at-rest
  • app.deployment-naming
  • app.dns-unpublished
  • app.edge-certificates
  • app.ingress-coverage
  • app.ingress-publish-posture
  • app.launcher-idp-address
  • app.older-renders
  • app.package-drift
  • app.pinned-package-fetchable
  • app.rendered-at-rest
  • app.staged-configs-at-rest
  • app.telemetry-export
  • app.unbound-plugs
  • audit.object-changes
  • backup.clone_staging
  • backup.scratch_volume
  • backup.staging_space
  • cluster.store-tiers
  • config.retired-environment
  • cron.fires
  • database.orphaned-dirs
  • database.orphaned-stacks
  • database.orphaned-vdeployments
  • database.storage
  • deployments.from-trust
  • deployments.missing-compose
  • dns.host-routing
  • docker.orphan-volumes
  • docker.registry-auth
  • doctor.self
  • events.cross-node-delivery
  • events.pending-broadcast
  • fabric.addressblock
  • fabric.forward-path
  • fabric.healthd-pins
  • fabric.host-setup
  • fabric.network-create
  • fabric.network-ranges
  • fabric.node-plane
  • health.certificates
  • health.queue-pressure
  • health.stuck-queues
  • health.watcher-trust
  • incus.api-latency
  • incus.exec-helper
  • incus.image-deletes
  • incus.image-replicas
  • incus.root-pool
  • incus.stray-dnsmasq
  • labelcheck.unknown
  • legacy.clusterfs-view
  • legacy.fabric-iptables
  • legacy.incus-base-form
  • legacy.licence-dir
  • legacy.mesh-iptables
  • legacy.move-journal
  • legacy.podman-parity
  • legacy.preset-sidecar
  • legacy.registry-auth
  • legacy.runtime-empty
  • legacy.state-mounts
  • legacy.vault-plaintext-pair
  • legacy.whole-project
  • mesh.mesh0
  • mesh.snapshot
  • node.podman-socket
  • node.podman-unit
  • node.self-clean
  • nodeagent.boot-gate
  • pods.clusterfs-whole-tree
  • pods.projections
  • pods.replica-projects
  • proxy.reconcile
  • repo.sync-state
  • resolver.receive-buffer
  • scheduler.disk_mark
  • state.container-mismatch
  • state.oom-kills
  • state.unrecorded-containers
  • storage.anonymous-volumes
  • storage.durable-on-node-disk
  • storage.fstab
  • storage.local
  • storage.lvm-drbd-filter
  • storage.thin-pools
  • storage.volume-mounts
  • system.auth-section
  • system.clusterfs-snapshot
  • system.directories
  • system.docker-address-pools
  • system.docker-daemon
  • system.gpu
  • tenant.former-baseline
  • underlay.pairs
  • underlay.udp
  • vault.autounlock_at_rest
  • vault.secrets
  • vault.status
  • vault.unlock_key_escrow
  • vault.unreadable_secrets
  • vip.drain

Every other check is cluster-scoped.

To add a diagnosis without touching the doctor module, publish a probe through the central registry from the module’s doctor.go and register it in the module’s init():

modules/myfeature/doctor.go
package myfeature
import (
"context"
"gitlab.com/bitspur/selfhosted-cloud/ce/core/registry"
)
func diagnoses() []registry.Diagnosis {
return []registry.Diagnosis{
{
Name: "myfeature.invariant",
Run: func(ctx context.Context) (status, hint string, err error) {
if !someInvariantHolds() {
return "warning: myfeature invariant broken",
"run `shc myfeature resync`", nil
}
return "ok", "", nil
},
},
}
}
// in register.go's init():
// registry.Diagnoses(ModuleName, diagnoses())

The daemon adopts every published probe at wire time (doctor.RegisterModuleDiagnoses in bridge.go), classifying "error:" / "warning:" status prefixes into finding severities. A probe whose check needs a daemon-resolved service registers via registry.DiagnosesWithDeps and resolves its *Service off the composition root per run (see vault/doctor.go). Bridged probes are inspection-only; the probe shape carries no prescribe callback.

  • Doctor: offline, inspects persisted state, can fix. Runs on demand from the CLI. No background loop.
  • Health: runtime, inspects liveness + heartbeats, reports only. Runs continuously as crons; feeds shc.status.get.

If you’re asking whether the deployment is up right now, use health. If you’re asking “is my state consistent?” use doctor.

  • shc doctor run [--fix] [--category X]: daemon-first: runs the sweep over POST /api/doctor/run when a daemon is reachable, and degrades to a local diagnose-only pass (with a warning; --fix refused) when none is. With --fix it starts the fix over POST /api/doctor/run_async and follows the operation to its end. See reference/cli/doctor
  • shc doctor follow <operation-id>: waits for a running fix again and prints its result.
  • shc doctor list: the registered diagnosis table, rendered locally.
  • POST /api/doctor/run: body {"category": "...", "fix": bool}, both optional; an empty body diagnoses only. On a cluster each node-scoped result carries node, see Scope. A body that asks for a fix runs it as the operation below and answers once it ends; the fix goes on if the caller hangs up first.
  • POST /api/doctor/run_async: the same body, which must ask for a fix. It answers the operation id at once. The fix runs on the daemon as operation doctor_fix on a context the caller cannot cancel, under the cluster’s operation claim doctor/fix, so shc operation ls lists it and a second fix is refused while it runs (X409717DCRRUN). Its steps and result follow on shc.operation.stream, and the daemon log has its start and end (FZ-7294).
  • GET /api/doctor/diagnoses: name/description/category/scope of every registered diagnosis.

Error codes live in the DCR band; the error reference is the generated list: X400921DCRUNK (no diagnosis by that name), X400922DCRNDS (the named diagnosis has no destructive fix), X500917DCROPF, X500918DCRNWR, X500923DCRNDY, X500924DCRNRA, X503919DCRKCD, X503920DCRNAV, and for a fix operation X409717DCRRUN (a fix already runs), X504849DCRFOL (the CLI lost the follow, the fix goes on) and X410565DCRGON (no node holds the operation any more).