Doctor module
Layer 4. Primitives. Primitives Checks have no owned record.
The doctor module is a diagnose-and-prescribe framework. It
runs a registered list of checks, each of which can optionally carry
a repair. The CLI entry point is shc doctor run: inspection-only
by default; repairs run only with an explicit --fix.
Unlike runtime health (shc health, shc hud), doctor is
intended for install-time, post-recovery, and offline
cleanup, the kind of issues that accumulate when
containers/volumes get cleaned up out-of-band or when a crashed
install leaves orphan state behind.
Source: modules/doctor/: framework.go holds the types
and the verdict tree, register.go seeds the core diagnoses from
diagnoses/, bridge.go adopts the probes sibling modules publish
through the central registry, and router.go exposes the HTTP
endpoints.
Responsibilities
Section titled “Responsibilities”- Framework types:
Diagnosis,Finding,DiagnosisResult, plus theSeverityandVerdictenums that callers render. - Runner:
Runneriterates the registered diagnoses, captures panics/errors intoSKIPPEDresults, and appliesprescribecallbacks only when fix is requested. - Registration: the doctor package’s own
init()seeds the core diagnoses; sibling modules (vault / storage / repo) publish slim(status, hint, err)probes viaregistry.Diagnoses/registry.DiagnosesWithDeps, and the daemon adopts those at wire time viadoctor.RegisterModuleDiagnoses(bridge.go). - Built-in diagnoses: ten checks across five categories covering Docker/Filesystem, DB-backed state, compose-rendered deployment dirs, DB↔container drift, and keycloak realms.
Diagnose / prescribe contract
Section titled “Diagnose / prescribe contract”Every Diagnosis is a record of:
type Diagnosis struct { Name string Description string Category string Diagnose DiagnoseFunc Prescribe Prescription // nil means inspection only}DiagnoseFunc is func(ctx context.Context) ([]Finding, error).
Zero findings is a pass. Prescribe is a remedy.Prescription. A
non-nil value carries a class: remedy.Benign, remedy.Mutating,
or remedy.Destructive. The runner sets Finding.Fixed when a
repair applied.
Verdict calculation
Section titled “Verdict calculation”The runner grades a result by its worst finding (Grade): info reads
INFO, warning reads WARN, error and critical read FAIL, and no
finding reads PASS. Under --fix (or {"fix": true} on the API) the
findings a prescription left unfixed decide the grade, and a result whose
every finding worse than info came back Fixed reads FIXED. A diagnose
error reads SKIPPED, and X200985DCRRTM reads INAPPLICABLE. The default
run is the preview: results whose skipped prescription could repair them
carry fixable=true. The ladder, what each verdict prints and the exit status
it gives are in reference/cli/doctor.
Exception containment
Section titled “Exception containment”DoctorRunner.Run isolates each diagnosis. A panic or an error from
Diagnose becomes one error-severity finding and a SKIPPED verdict
for that check. The rest of the sweep still runs. X200985DCRRTM is the
one error that is not a skip: the check reads docker on a node whose
runtime runs none, so its verdict is INAPPLICABLE with no finding, and
the condition sweep counts it as observed. A prescribe error
is appended as an error-severity finding, so that check reads FAIL.
Built-in inventory
Section titled “Built-in inventory”Registered from modules/doctor/diagnoses/:
| Category | Diagnosis name | Prescribe? | What it checks |
|---|---|---|---|
system | system.docker-daemon | no | Docker daemon reachable |
system | system.docker-address-pools | yes | default-address-pools is provisioned and has headroom (never restarts dockerd) |
system | system.directories | yes | Required SHC directories exist and are writable |
system | system.gpu | no | An NVIDIA GPU the driver shows that the node’s runtime cannot pass to a container, and what the node lacks (the container toolkit, a CDI spec) |
system | system.auth-section | no | This node’s clusterfs auth section is fresh: a proof older than five minutes, or of another digest than it serves, makes the node’s forward-auth answer every gated request 503 stale_replica, and the finding names the node |
system | system.clusterfs-snapshot | no | The store snapshot each clusterfs build copies into memory on this node stays under 64 MiB: past it every build on every node pays that copy, and the finding names the node, the rows and the size |
database | database.orphaned-stacks | yes | DB stack rows whose deployment dir is gone |
database | database.orphaned-dirs | no | Deployment dirs with no matching DB row |
database | database.stale-status | yes | status / desired words disagree with running state |
database | database.storage | no | Database file present, sized sanely, writable |
state | state.container-mismatch | yes | DB says running but no container (or vice versa) |
state | state.oom-kills | no | Containers on this node the kernel OOM killer reaped. The kernel log names every kill at a container’s own memory limit in the last 24 hours, worker kills that leave the container running and incus crash loops included, with the tenant, deployment, service, limit, peak and count. An OOM exit the node recorded is reported when the log no longer holds it, and is the only record of a container with no limit of its own that the host ran out of memory under. Two or more kills fail |
state | state.unrecorded-containers | no | SHC-labelled containers on this node with no deployment row or no service row, split into debris (docker rm), rowless declared services (shc update) and undeclared containers |
deployments | deployments.missing-compose | yes | Stack has a template but no rendered compose.yaml |
deployments | deployments.unhealthy | no | A deployment wanted running reads degraded, failed or stopped for a sweep (deployment.Unhealthy) |
deployments | deployments.scheduled-jobs | no | A deployment wanted running whose scheduled standalone job failed its last finished run, named with the job, the run, when and its error; the next passing run clears it (deployment.JobFailed) |
deployments | deployments.replicas-down | no | A replica wanted running stood assigned, starting, unready or failed on its node for 10 minutes, named with its node, state and reason |
deployments | deployments.replicas-flapping | no | A replica whose readiness flipped 4 times in 10 minutes, named with its node and whether routing holds it out (deployment.ReplicaFlapping) |
deployments | deployments.from-trust | no | A deployment running the app a --from put in whose sidecar does not record who chose that source, so it renders with its own source’s trust; the finding gives the shc update --from that records it |
keycloak | keycloak.realms | yes | Every tenant has a keycloak realm |
doctor | doctor.self | no | The doctor’s own dependency seams are wired |
On the daemon the table additionally carries the module probes
adopted from the central registry (vault.status, vault.secrets,
vault.crypto_format, storage.local, repo.sync-state):
inspection-only entries whose category is the publishing module’s
name.
The categories are free-form strings; the CLI’s --category/-c
flag filters by exact match.
Every check states its scope in code (Scope on the diagnosis):
cluster when its answer is the same from every node, node when it
reads the node that runs it (disk, container engine, network, local
services). shc doctor ls prints it in its SCOPE column.
shc doctor run on a cluster reports the cluster. The node it reaches
runs each cluster check once and its own node checks, and asks every
other roster node at once over the internal RPC doctor.node_run for
its node checks in the run’s category. A node check prints one row per
node, the node’s name in the last column:
$ shc doctor run -c dns[fail] dns.host-routing incus01 error: no route to 10.236.0.1[pass] dns.host-routing incus02[skipped] doctor.node-checks incus03 error: incus03 did not answer: connection refusedA node the roster reads unreachable is not dialled, and it and a node
that does not answer within 90 seconds each print one skipped
doctor.node-checks row saying why. When the roster cannot be read,
one doctor.node-checks row says so and the node checks answer for the
node reached alone. The other nodes answer diagnose-only: a node
applies its own fixes, so another node’s row carries its fix withheld
with the command to run there. The exit code and the summary count every
row on every node.
The condition sweep runs on the leader. It runs the cluster checks once,
runs the node checks for itself, and asks every other node for its node
checks over the internal RPC doctor.node_checks. A node-scoped
condition’s subject carries the node id, so each node holds its own
condition and its own “since”. A node that does not answer, or that the
roster reads unreachable, reads check.Unobservable for each node check
and its open node conditions turn Unknown; none of them clears until the
node answers again. A node that leaves the roster takes its node
conditions with it.
Node-scoped checks
Section titled “Node-scoped checks”api.public-listenerapp.appyaml-driftapp.backup-v0121-meaningapp.cert-homes-at-restapp.deployment-namingapp.dns-unpublishedapp.edge-certificatesapp.ingress-coverageapp.ingress-publish-postureapp.launcher-idp-addressapp.older-rendersapp.package-driftapp.pinned-package-fetchableapp.rendered-at-restapp.staged-configs-at-restapp.telemetry-exportapp.unbound-plugsaudit.object-changesbackup.clone_stagingbackup.scratch_volumebackup.staging_spacecluster.store-tiersconfig.retired-environmentcron.firesdatabase.orphaned-dirsdatabase.orphaned-stacksdatabase.orphaned-vdeploymentsdatabase.storagedeployments.from-trustdeployments.missing-composedns.host-routingdocker.orphan-volumesdocker.registry-authdoctor.selfevents.cross-node-deliveryevents.pending-broadcastfabric.addressblockfabric.forward-pathfabric.healthd-pinsfabric.host-setupfabric.network-createfabric.network-rangesfabric.node-planehealth.certificateshealth.queue-pressurehealth.stuck-queueshealth.watcher-trustincus.api-latencyincus.exec-helperincus.image-deletesincus.image-replicasincus.root-poolincus.stray-dnsmasqlabelcheck.unknownlegacy.clusterfs-viewlegacy.fabric-iptableslegacy.incus-base-formlegacy.licence-dirlegacy.mesh-iptableslegacy.move-journallegacy.podman-paritylegacy.preset-sidecarlegacy.registry-authlegacy.runtime-emptylegacy.state-mountslegacy.vault-plaintext-pairlegacy.whole-projectmesh.mesh0mesh.snapshotnode.podman-socketnode.podman-unitnode.self-cleannodeagent.boot-gatepods.clusterfs-whole-treepods.projectionspods.replica-projectsproxy.reconcilerepo.sync-stateresolver.receive-bufferscheduler.disk_markstate.container-mismatchstate.oom-killsstate.unrecorded-containersstorage.anonymous-volumesstorage.durable-on-node-diskstorage.fstabstorage.localstorage.lvm-drbd-filterstorage.thin-poolsstorage.volume-mountssystem.auth-sectionsystem.clusterfs-snapshotsystem.directoriessystem.docker-address-poolssystem.docker-daemonsystem.gputenant.former-baselineunderlay.pairsunderlay.udpvault.autounlock_at_restvault.secretsvault.statusvault.unlock_key_escrowvault.unreadable_secretsvip.drain
Every other check is cluster-scoped.
Extending doctor from another module
Section titled “Extending doctor from another module”To add a diagnosis without touching the doctor module, publish a
probe through the central registry from the module’s doctor.go and
register it in the module’s init():
package myfeature
import ( "context"
"gitlab.com/bitspur/selfhosted-cloud/ce/core/registry")
func diagnoses() []registry.Diagnosis { return []registry.Diagnosis{ { Name: "myfeature.invariant", Run: func(ctx context.Context) (status, hint string, err error) { if !someInvariantHolds() { return "warning: myfeature invariant broken", "run `shc myfeature resync`", nil } return "ok", "", nil }, }, }}
// in register.go's init():// registry.Diagnoses(ModuleName, diagnoses())The daemon adopts every published probe at wire time
(doctor.RegisterModuleDiagnoses in bridge.go), classifying
"error:" / "warning:" status prefixes into finding severities. A
probe whose check needs a daemon-resolved service registers via
registry.DiagnosesWithDeps and resolves its *Service off the
composition root per run (see vault/doctor.go). Bridged probes are
inspection-only; the probe shape carries no prescribe callback.
Doctor vs. health
Section titled “Doctor vs. health”- Doctor: offline, inspects persisted state, can fix. Runs on demand from the CLI. No background loop.
- Health: runtime, inspects liveness + heartbeats,
reports only. Runs continuously as crons; feeds
shc.status.get.
If you’re asking whether the deployment is up right now, use health. If you’re asking “is my state consistent?” use doctor.
CLI + HTTP surface
Section titled “CLI + HTTP surface”shc doctor run [--fix] [--category X]: daemon-first: runs the sweep overPOST /api/doctor/runwhen a daemon is reachable, and degrades to a local diagnose-only pass (with a warning;--fixrefused) when none is. With--fixit starts the fix overPOST /api/doctor/run_asyncand follows the operation to its end. See reference/cli/doctorshc doctor follow <operation-id>: waits for a running fix again and prints its result.shc doctor list: the registered diagnosis table, rendered locally.POST /api/doctor/run: body{"category": "...", "fix": bool}, both optional; an empty body diagnoses only. On a cluster each node-scoped result carriesnode, see Scope. A body that asks for a fix runs it as the operation below and answers once it ends; the fix goes on if the caller hangs up first.POST /api/doctor/run_async: the same body, which must ask for a fix. It answers the operation id at once. The fix runs on the daemon as operationdoctor_fixon a context the caller cannot cancel, under the cluster’s operation claimdoctor/fix, soshc operation lslists it and a second fix is refused while it runs (X409717DCRRUN). Its steps and result follow onshc.operation.stream, and the daemon log has its start and end (FZ-7294).GET /api/doctor/diagnoses: name/description/category/scope of every registered diagnosis.
Error codes live in the DCR band; the
error reference is the generated
list: X400921DCRUNK (no diagnosis by that name), X400922DCRNDS
(the named diagnosis has no destructive fix), X500917DCROPF,
X500918DCRNWR, X500923DCRNDY, X500924DCRNRA, X503919DCRKCD,
X503920DCRNAV, and for a fix operation X409717DCRRUN (a fix already
runs), X504849DCRFOL (the CLI lost the follow, the fix goes on) and
X410565DCRGON (no node holds the operation any more).
Related
Section titled “Related”- Health module: runtime liveness, not state drift
- App module: owns
DeploymentModelthat several diagnoses reconcile against - Reference: doctor CLI