Health module
Layer 11. Lifecycle. Lifecycle It reconciles a running deployment.
Source: modules/health/.
At a glance
Section titled “At a glance”| CLI | shc status |
| HTTP endpoints | see the route reference |
| Cross-module deps | app, backup, docker, job, metrics, network, node, vault |
Source layout
Section titled “Source layout”modules/health/├── commands/├── events.go├── poll_deployments.go├── poll_nodes.go├── router.go # /health, /ready endpoints├── state.go # module-level HealthState singleton├── tasks.go└── types.go # HealthState, NodeHealth, DeploymentHealthResponsibilities
Section titled “Responsibilities”Three concerns, all rooted in the health module:
- Runtime health endpoints: liveness, readiness, and a detailed
health report for load balancers,
shc status, and monitoring. - Cron tiers: the structured periodic work that polls the cluster, reconciles degraded deployments, and purges old rows.
- The
HealthStatesingleton: the module-level snapshot of cluster state that crons write to and the HUD / endpoints read from.
The module owns no subsystem lifecycle; HealthState is a plain
module-level object decoupled from subsystem start/stop.
Cron tier model
Section titled “Cron tier model”All periodic work runs through three leader-only cron tiers plus a set of DB-tracked maintenance cron jobs. Every cron uses skip-if-running: if the previous invocation has not completed, the next tick is silently skipped.
Tier summary
Section titled “Tier summary”| Cron | Type | Default | Config path |
|---|---|---|---|
| Tick | rpc cron | 15 s | timing.tick_interval |
| Poll | rpc cron | 1 min | timing.poll_interval |
| Reconcile | rpc cron | 1 min | timing.reconcile_interval |
| Cleanup | task cron | 1 h | maintenance.cleanup.interval |
| Disk scan | task cron | 1 h | maintenance.disk_scan.interval |
| Usage reconcile | task cron | 6 h | maintenance.usage_reconcile.interval |
| Disk auto-prune | task cron | 1 h | maintenance.disk_auto_prune.interval |
| Repo sync | task cron | 6 h | maintenance.repo_sync.interval |
| Retention cleanup | task cron | daily 02:00 | maintenance.retention_cleanup.schedule |
All intervals are config-driven via function references resolved at scheduler start; see cron in the communication-patterns doc.
Tick: 15 s, leader-only
Section titled “Tick: 15 s, leader-only”Fast, lightweight data gathering. Runs sequentially within one tick:
node_metrics: broadcast gather, each node returns system metrics; leader writes toHealthState.container_metrics: broadcast gather, each node returns docker stats.global_diagnostics: local leader query for task/worker/cron counts.quota_metrics: local leader collection of quota gauges.vault_check: broadcast fire-and-forget; if any node reports locked and the leader holds the password in memory, re-broadcast unlock (see Vault unlock flow).
Poll: 1 min, leader-only
Section titled “Poll: 1 min, leader-only”Moderate-weight state sync:
deployment_sync: broadcastdocker compose psto every node; leader writes container/service state to DB and toHealthState.retention_config_check: leader-local audit of retention policies.
The deployment rollup: what “healthy N/N” actually claims
Section titled “The deployment rollup: what “healthy N/N” actually claims”deployment_sync (leader) and the follower-side read-path derivation
(Service.Status when no local poll has run) share ONE fold:
deriveOneDeployment in modules/health/derive.go. The evidence it
consults, per container:
| Fact | Source | Used for |
|---|---|---|
status | recorder-written container row (raft-replicated) | readiness |
health | same | readiness; blank ⇒ unprobed |
| SHC container status | derived at the seam from exit code + restart policy | a restart “no” service that ended 0 counts as ready |
exit_cause / oom_kill_count / oom_memory_limit_bytes | same row | the reason line |
updated_at | same row | staleness (a frozen row = a gone container) |
RestartCount / State.StartedAt | node-local metrics.StatsCache (2 s tick, fanned in cluster-wide) | liveness |
The last row is the newest and closes a measured defect: shc status
reported mailcow healthy 18/18 while watchdog-mailcow had restarted
117 times. Every fact above the line describes the container at an
instant, and a crash-looping container is “Up” most instants, so a
loop and a steady process were indistinguishable to the fold. (The
daemon was not ignorant of it: the reason column read
watchdog-mailcow stopped, cause unknown beside the healthy verdict.)
The rule (modules/health/restart_signal.go, IsFlapping):
a container is FLAPPING when
RestartCount >= 3andnow - State.StartedAt < 10m.
Why both halves:
RestartCountis cumulative and never resets, so it alone cannot say whether the restarts are happening now: a container with 117 lifetime restarts that has been solid for a week would stay red forever, and a permanently-red signal is one nobody reads.State.StartedAtsupplies the “now” without any remembered history. For a container under a restart policy the age of the current run is the observed inter-restart interval. A window-delta rate would need samples, so it would be blind after every daemon restart and every leader change, and it could not see a loop with a period longer than the window.- 3 excludes the ordinary deploy race (an app losing the connect to its database once or twice before it settles), the most common restart shape on a healthy estate. Docker’s restart backoff doubles from 100 ms, so three restarts is under a second of backoff: looping, not recovering.
- 10 min is longer than a slow-but-legitimate start (a database
doing crash recovery) and short enough that “flapping” means now. It
is also the self-clearing mechanism: nothing resets
RestartCount, so the settle window is the only thing that can return a deployment to healthy.
Effect on the verdict. A flapping service is excluded from
services_healthy (mailcow now reads 17/18, not 18/18), and the
reason names it: watchdog-mailcow restarting repeatedly (117 restarts, up 8s). A deployment whose only fault is flapping is clamped to
degraded, never unhealthy; unhealthy means every container is
down, and a looping container is up part of the time. When something is
genuinely down alongside it, unhealthy stands.
It fails open. No cache entry (cold daemon, collectors not warm, a
peer whose fan-in has not landed) or a zero StartedAt means no
verdict, and the rollup answers exactly as it did before. The signal
does not feed the reconciler: ReconcileDegradedStacks reads the
deployment row status via ListBrokenDeployments, not HealthState,
so a flapping verdict cannot drive an auto-restart loop that would feed
itself more restarts.
Unprobed services: a running-state claim is not a liveness claim
Section titled “Unprobed services: a running-state claim is not a liveness claim”A RUNNING container whose observer reports no health token declares no
healthcheck; docker emits healthy / unhealthy / health: starting
for every container that has one. It has therefore passed nothing.
Measured on huly: 12/12 healthy where EIGHT of the twelve services
declared no probe at all.
The rollup now counts them (services_unprobed on the wire, omitted
when zero) and shc status names them under the table:
3 services counted healthy declare no healthcheck; that is arunning-state claim, not a liveness claim: huly (8), plane (2), ...The verdict is deliberately unchanged. Subtracting unprobed services
from services_healthy is the strictly honest fold and it would flip a
large share of both production estates to degraded in one release, for
a condition no operator can fix from the status surface; the fix is a
healthcheck in the app (as huly’s was). Publishing the number
puts the pressure where it can be acted on without an alarm storm.
Note also that a probe existing is not sufficient: huly’s original socket-listening check reported LISTENING for every SIGSTOP-frozen process, so “the socket is open” is not liveness either. The unprobed count measures probe presence, not probe quality.
Reconcile: 1 min, leader-only
Section titled “Reconcile: 1 min, leader-only”Self-healing for degraded deployments plus infrastructure enforcement. Broadcast fire-and-forget; each node acts on its own local state.
Per-node flow:
- Read in-memory
HealthStatefor this node’s deployments. - If any deployment is degraded, do a live
docker compose psre-check before acting; never reconcile on stale in-memory state alone. - If still degraded:
- Skip if already in the resolving set for this deployment.
- Skip if cooldown has not elapsed since last attempt.
- Apply exponential backoff on repeated failures.
- Add to resolving set → run
docker compose up -d→ remove on completion.
- All cooldown / backoff state is in-memory only and resets on process restart.
Infrastructure enforcement (same cron): Traefik port-binding check and restart, OpenObserve health check, docker network creation.
Maintenance crons
Section titled “Maintenance crons”Run via task cron, so each invocation creates a TaskModel row
and is audit-tracked.
The Cleanup job also does task purging:
- Completed tasks older than
maintenance.completed_task_retention(default 24 h) are bulk-deleted. - Failed tasks older than
maintenance.failed_task_retention(default 7 d) are bulk-deleted.
This is a single bulk DELETE; the retention defaults
keep failed rows longer for debugging.
HealthState singleton
Section titled “HealthState singleton”Lives at modules/health/state.go as a module-level instance of
the HealthState struct:
type HealthState struct { Nodes map[string]*NodeHealth Deployments map[string]*DeploymentHealth LastPoll time.Time}Writers: the Tick and Poll crons (node_metrics, container_metrics,
deployment_sync). Readers: the HUD collector, shc status, and the
detailed shc.status.get method.
Alongside HealthState, the same module holds:
_cron_heartbeats: dict[str, datetime]: last successful completion per cron (read by the HUD for heartbeat display).- Reconcile backoff state (cooldown dict, resolving set) also in-memory only.
Vault unlock flow
Section titled “Vault unlock flow”Unlock: user POSTs to shc.vault.unlock with the password:
- Request is routed to the leader (existing behavior).
- Leader calls
unlock_vault(password)locally; password held in memory only. - Leader calls the
vault.unlockRPC function on every other node over the mTLS internal API. - Each node unlocks locally.
Ongoing enforcement runs from the Tick cron’s vault_check step:
leader broadcasts a vault-status check; any node reporting locked
triggers a re-broadcast of the unlock if the leader still holds
the password. On leader restart, the leader first tries the
configured env var / password manager; if neither yields a password,
nodes stay locked until a human unlocks again.
Security constraints:
- There is no endpoint to read the vault password, only write (unlock).
- The password is only transmitted over the mTLS internal API (port 4003).
Performance at scale
Section titled “Performance at scale”| Metric | 1 node | 10 nodes | 50 nodes |
|---|---|---|---|
| Tick broadcast RPCs / min | 4 (self) | 40 | 200 |
| Poll broadcast RPCs / min | 1 | 10 | 50 |
| Reconcile broadcast RPCs / min | 1 | 10 | 50 |
| Internal-API calls / min (total) | ~6 | ~60 | ~300 |
Broadcasts run concurrently through goroutines and errgroup in
RpcTransport.FanOut; 50 nodes = 5 calls/s, comfortably within
mTLS connection capacity.
See also
Section titled “See also”- CLI reference:
shc status - Communication patterns: RPC and task systems
hudmodule: consumesHealthStatevia the internal API.vaultmodule