Skip to content
SHC Docs

SHC status states

Layer 6. Services. Services

SHC derives deployment status and health from runtime container observations. The same derivation feeds service progress and deployment state. The node agent applies restart policy under the scheduler’s assignments and leases.


SHC tracks status and health at three layers:

Deployment
└── Services
└── Containers

Each layer has:

  • Status: Lifecycle state (running, stopped, degraded, etc.)
  • Health: Application health (healthy, unhealthy, etc.)

Status and health are independent; a container can be running but unhealthy.


LayerFieldsDerived From
Deploymentstatus, healthServices
Servicestatus, health, desired_replicas, restart_policyContainers
Containershc_status, health, status, exit_codeDocker/Podman

Key principle: Lower layers feed into higher layers. Container states derive service states, which derive deployment states.


VarMeaning
dsDocker container status (raw)
dhDocker container health (raw)
ecExit code
rpRestart policy
drDesired replicas
csSHC container status (derived)
chSHC container health (derived)
ssSHC service status
shSHC service health
rtRuntime selected for the node

LayerFieldValues
Docker Containerstatus (raw)created, running, paused, restarting, exited, dead
Docker Containerhealth (raw)none, "" (no check), starting, healthy, unhealthy, unknown
SHC Containerstatusstarting, running, paused, complete, failed, degraded, unknown
SHC Containerhealth (derived)healthy, unhealthy, starting, none, unknown
SHC Servicestatusstopped, starting, running, restarting, paused, complete, failed, degraded
SHC Servicehealthunknown, unhealthy, starting, healthy, none
SHC Deploymentstatus (operational)queued, created, installing, starting, stopping, updating, removing
SHC Deploymentstatus (derived)running, paused, degraded, complete, failed, stopped; read only, never stored: moving (see Moving)
SHC Deploymenthealthunknown, unhealthy, starting, healthy, none

For status derivation, NormalizeRestartPolicy in modules/app/models.go normalizes the authored policy:

InputStatus policyMeaning
nonoAn exit is expected
on-failureon-failureRetry a failed job
alwaysalwaysKeep the service running
unless-stoppedalwaysDerive daemon status

The render writes restart: "no" for the runtime and saves the authored policy in shc.restart.* labels. The node agent reads those labels. Compose deploy.restart_policy.condition is translated before this step; see compose keys.


Docker provides these container states. SHC stores them exactly as Docker reports:

StateDescription
createdContainer exists, never started
runningContainer is running
pausedContainer processes suspended
restartingContainer is restarting
exitedContainer stopped
deadContainer failed to stop properly

Docker provides health check status. SHC stores the raw value it reads, as Docker reports it:

HealthDescription
""Empty; the container declares no HEALTHCHECK (this is what the engine actually returns; not the literal word none)
noneSynonym some surfaces emit for “no health check defined”
startingHealth check defined but not yet passed
healthyHealth check passing
unhealthyHealth check failing
unknownHealth could not be read

Note: The engine reports "" (empty string) for a container with no HEALTHCHECK, and the recorders write Docker’s own word for that state, none (app.NoHealthcheck), into health for a running container. A row the recorders never stamped reads "" (NULL). Every reader treats "" and none identically: running with no check is ready. unknown is distinct from both: it means “couldn’t determine,” not “no check.”

The raw docker health string is not a passthrough. SHC derives a canonical container-health word from both the raw health and the raw docker status, because health is only meaningful while a container is live. The derivation is DeriveContainerHealth(rawHealth, dockerStatus) and produces one of five canonical values:

HealthMeaning
healthyLive container whose check is passing or that declares no check at all (a running container with nothing left to check is as healthy as it can report)
unhealthyLive container whose health check is failing
startingLive container whose health check has not yet passed
noneContainer is not live (created / exited / complete / failed / dead / paused) → health is not applicable. This is a marker, not a displayed word; the UI shows such a container’s STATUS instead. A running container is never none.
unknownHealth could not be determined. An unrecognised engine value also yields unknown.

Key invariants:

  • A running container is never none; a running no-check container is healthy (“live, nothing left to wait on”). The model emits the final word once and every renderer trusts it; there is no per-renderer “promote none → healthy” logic.
  • none is the not-applicable marker for non-running containers; a completed job never reports a ghost healthy; its health derives to none regardless of any stale raw value, and the display shows its STATUS instead. none never displays as a health word.
  • dh=unknown yields unknown before the container status is consulted.
Docker StatusRaw HealthSHC Container Health
anyunknownunknown (undeterminable wins)
runninghealthyhealthy
runningunhealthyunhealthy
runningstartingstarting
running"" / nonehealthy (live + no check = healthy)
running(unrecognized)unknown
created / exited / complete / failed / dead / pausedany (except unknown)none (not-applicable; display shows STATUS)
flowchart LR
D0{"dh (raw)"}
D0-->|unknown|UNKNOWN[unknown]
D0-->|not unknown|D1{ds}
D1-->|not running|NONE[none]
D1-->|running|D2{dh}
D2-->|unhealthy|UNHEALTHY[unhealthy]
D2-->|starting|STARTING[starting]
D2-->|healthy|HEALTHY[healthy]
D2-->|"empty or none"|HEALTHY
D2-->|unrecognized|UNKNOWN

The order is load-bearing: dh=unknown is checked first (before the status), then a non-running status forces none, and only a running container consults the remaining raw health words. A running container therefore lands on healthy / unhealthy / starting / unknown but never none; a non-running one lands on none unless its health was already unreadable (unknown).

SHC interprets Docker status based on exit code and restart policy to produce a semantic status:

StatusMeaning
startingContainer initializing or restarting
runningContainer running and healthy
pausedContainer intentionally paused
completeJob finished successfully (exit 0, job mode)
failedJob failed (exit != 0, no retry)
degradedNot running when expected

Note: The status derivation consumes the raw docker health (dh), not the derived container health (ch) above; its truth table stays documented in raw terms. Both the raw "" (no check) and none map identically to running here, and any unrecognized health word also folds to running (so a new docker health value never silently degrades every container).

flowchart LR
D1{ds}
D1-->|created|DC{"Active converge?"}
DC-->|yes|STARTING[starting]
DC-->|no|DEGRADED[degraded]
D1-->|restarting|STARTING
D1-->|dead|DEGRADED[degraded]
D1-->|paused|PAUSED[paused]
D1-->|running|D2{dh}
D2-->|starting|STARTING
D2-->|healthy|RUNNING[running]
D2-->|"empty or none"|RUNNING
D2-->|unhealthy|DEGRADED
D2-->|"unrecognized"|RUNNING
D1-->|exited|D3{ec}
D3-->|0|D4{rp}
D4-->|no|COMPLETE[complete]
D4-->|on-failure|COMPLETE
D4-->|always|DEGRADED
D3-->|!=0|D5{rp}
D5-->|no|FAILED[failed]
D5-->|on-failure|DEGRADED
D5-->|always|DEGRADED
D3-->|NULL|D6{rp}
D6-->|no|UNKNOWN[unknown]
D6-->|on-failure|UNKNOWN
D6-->|always|DEGRADED
Docker StatusExit CodeHealthRestart PolicySHC Status
created during active converge-anyanystarting
created outside active converge-anyanydegraded
restarting-anyanystarting
running-startinganystarting
running-healthyanyrunning
running-"" / noneanyrunning
running-unhealthyanydegraded
running-(unrecognized)anyrunning
paused-anyanypaused
exited0anynocomplete
exited0anyon-failurecomplete
exited0anyalwaysdegraded
exited!=0anynofailed
exited!=0anyon-failuredegraded
exited!=0anyalwaysdegraded
exitedNULLanynounknown
exitedNULLanyon-failureunknown
exitedNULLanyalwaysdegraded
exitedany, or NULLanydefault (no restart: key)degraded
deadanyanyanydegraded

A NULL exit code is never read as 0: under no and on-failure only the code tells complete from failed, so the row reads unknown, which the service tier counts in no bucket (degraded) and the health rollup never counts as ready.

Every surface reads a container from the same two facts, its exit code and the restart policy its package authored. The Deployment listing, the Deployment read and the stack status read fold the cluster’s replicated container rows; this node’s engine probe stands in only for a deployment the rows cannot answer (no fresh row, or an operation in flight), and it reads the exit code and the shc.restart.policy and shc.restart.defaulted labels off each container. A run-once that exited 0 therefore reads complete on every route, and one that exited non-zero reads failed, its deployment degraded with the reason traefik-init failed, so it cannot be running.


Service status is derived from container statuses, desired replica count, and restart policy:

StatusMeaning
stoppedIntentionally scaled to zero (desired_replicas = 0)
pausedEvery container is paused (an intentional full-service suspension, e.g. a backup quiesce). A partial pause is treated as alive and reads running
startingContainers starting up (or no containers observed yet)
runningEnough replicas alive: count(running + paused) >= desired_replicas
restartingA container is mid-restart
completeAll jobs finished successfully (job mode)
failedAll jobs failed (job mode)
degradedPartial availability or unexpected state

Paused counts as alive for the running rule: running + paused >= desired_replicas reads running, so a short-lived rolling pause (some containers paused, the rest up) does not flap the status. Only when all containers are paused does the service surface as paused. paused is a derived status.

flowchart LR
D1{dr}
D1-->|= 0|STOPPED[stopped]
D1-->|> 0|DP{"all c : cs=paused (total>0)"}
DP-->|yes|PAUSED[paused]
DP-->|no|D2{"n(running + paused) >= dr"}
D2-->|yes|RUNNING[running]
D2-->|no|DR{"any c : cs=restarting"}
DR-->|yes|RESTARTING[restarting]
DR-->|no|D3{"any c : cs=starting or n(c)=0"}
D3-->|yes|STARTING[starting]
D3-->|no|D4{rp}
D4-->|no / on-failure|D5{"all c : cs=complete"}
D5-->|yes|COMPLETE[complete]
D5-->|no|D6{"all c : cs=failed"}
D6-->|yes|FAILED[failed]
D6-->|no|DEGRADED[degraded]
D4-->|always|DEGRADED
  1. If desired_replicas = 0 -> stopped
  2. If total > 0 and all(cs = paused) -> paused (whole-service suspension; checked before the running rule so an all-paused service does not read running)
  3. If count(cs = running) + count(cs = paused) >= desired_replicas -> running (paused counts as alive; a partial pause lands here)
  4. If any(cs = restarting) -> restarting
  5. If any(cs = starting) or total = 0 -> starting
  6. If job mode (rp = no or on-failure):
    • If all(cs = complete) -> complete
    • If all(cs = failed) -> failed
  7. Else -> degraded

Note: Job completion/failure (step 6) requires at least one container and the job restart policy. If a service has desired_replicas > 0 but no containers exist yet, step 5 (total = 0) makes the status starting. A daemon-mode (rp = always) service that reaches step 6 falls through to degraded.

Service health is derived from the per-container (derived) health using “worst wins” priority:

HealthMeaning
unknownNo containers, or all containers were undeterminable (ch = unknown)
unhealthyAny container unhealthy
startingAny container health starting
healthyAny container healthy
noneAll containers non-live / not-applicable (ch = none)

After checking unhealthy, starting and healthy, any unknown container health yields unknown. The result is none only when no other signal exists.

flowchart LR
D1{"n(c)"}
D1-->|= 0|UNKNOWN[unknown]
D1-->|> 0|D2{"any c : ch=unhealthy"}
D2-->|yes|UNHEALTHY[unhealthy]
D2-->|no|D3{"any c : ch=starting"}
D3-->|yes|STARTING[starting]
D3-->|no|D4{"any c : ch=healthy"}
D4-->|yes|HEALTHY[healthy]
D4-->|no|D5{"any c : ch=unknown"}
D5-->|yes|UNKNOWN
D5-->|no|NONE[none]

Deployment status has two categories:

Operational states: Set by user commands, block derived updates:

StatusTrigger
createdshc install
startingshc start or compose up
stoppingshc stop
updatingshc update
removingshc uninstall

Derived states: Computed from service statuses when not in operational state (the derivation never overwrites an operational state; callers MUST check IsOperational() first):

StatusMeaning
runningAll services running (or a partial pause / all-running-or-complete mix)
pausedEvery service is paused (whole-deployment suspension, e.g. a backup quiesce). A partial pause reads running; a paused mix that also has a bad state reads degraded
degradedAny service degraded
completeAll services complete (job deployment)
failedAll services failed
stoppedAll services stopped

paused at the deployment tier surfaces only when all services are paused. The check runs first, so an all-paused deployment does not read running. A partial pause (some services running or done, the rest paused, no bad state) is treated as alive and reads running. A pause mixed with any degraded or failed service reads degraded. paused is derived at both tiers.

A replica move takes a replica down on purpose. A replica that carries a volume is stopped while its data is copied to the node it goes to; one that carries none is brought up there beside its copy, and the copy then goes. shc update --where and --prefer, a node drain and shc move of a replica all move replicas this way. For that while the deployment is short a ready replica by plan, and that is not a fault.

So shc ls, shc status and their -o json read such a deployment moving, never degraded:

WordWhen
movingA planned move holds a replica down, and every container or replica that is not ready is that replica, on the node it leaves or the node it goes to
degradedAnything else is down: a replica lost outside a plan (its container killed, its node gone), or a service no move takes

The row’s reason names each replica being moved, the node it leaves, the node it goes to, since when, and the plan’s downtime when the plan made one:

counter-r0 moving from far10 to home32 since 2026-10-09T18:31:04Z, about 32s down
api-r0 moving from far10 to home32 since 2026-10-09T18:30:51Z, its copy serving until it is up
counter-r0 moved from far10 to home32, starting there

The replica’s own row in the shc status -s replica table reads moving too, and the services that are not being moved read as they are. -o json carries each move under moving. A move that has landed is read for two minutes more, while the replica comes up on its new node and its old copy goes; the deployment reads healthy again once it is up. A move that fails puts the replica back and stops reading moving at once.

A whole move takes the deployment down as a whole: shc move of a deployment that runs on one node, a drain’s whole moves, and a move with a rename or into another tenant or environment stop it, bring it up on the target and tear the source down. For that while its row reads moving too, from whatever word the move leaves it in (stopped, unknown, starting, degraded, unhealthy), and every replica it took reads moving. A move under another name or scope reads moving on the source row and on the target row; the target reads healthy once it is up. A running or failed row keeps its word, and a deployment lost outside a plan keeps its fault word.

the deployment moving from far10 to home32 since 2026-10-09T20:30:00Z, about 2m4s down
the deployment moved from far10 to home32, starting there

The move records the replica it takes, or the deployment it moves as a whole, in the replicated store before it takes it down, so every node answers the same. The record counts only while the move holds the deployment’s relocation lease: a move whose process died stops reading moving within the lease’s time to live. moving is read at the answer and never stored, so the stored status and everything that gates on it are unchanged.

When the deployment is not in an operational state, its status is derived from the per-service statuses (first matching rule wins):

flowchart LR
S0{"n(s)"}
S0-->|= 0|STOPPED0[stopped]
S0-->|> 0|DP{"all s : ss=paused"}
DP-->|yes|PAUSED[paused]
DP-->|no|D2{"all s : ss=running"}
D2-->|yes|RUNNING[running]
D2-->|no|D3{"any s : ss=starting"}
D3-->|yes|STARTING0[starting]
D3-->|no|D4{"all s : ss=complete"}
D4-->|yes|COMPLETE[complete]
D4-->|no|D5{"all s : ss=failed"}
D5-->|yes|FAILED[failed]
D5-->|no|D6{"all s : ss=stopped"}
D6-->|yes|STOPPED[stopped]
D6-->|no|D7{"running+complete = total and running>0"}
D7-->|yes|RUNNING
D7-->|no|D8{"running+paused+complete = total and (running>0 or paused>0)"}
D8-->|yes|RUNNING
D8-->|no|D9{"any s : ss=degraded, or any running/failed/paused"}
D9-->|yes|DEGRADED[degraded]
D9-->|no|D10{"any s : ss=complete"}
D10-->|yes|COMPLETE
D10-->|no|STOPPED

The two paused-aware rungs (the all-paused check at the top, and the running+paused+complete partial-pause rung) are what keep a rolling backup pause reading running rather than degraded, while a pause mixed with a bad state still drops to degraded.

Deployment health uses the service tier’s priority order. unknown follows healthy; an all-unknown deployment reports unknown:

HealthMeaning
unknownNo services, or all service health was undeterminable (sh = unknown)
unhealthyAny service unhealthy
startingAny service health starting
healthyAny service healthy
noneAll services non-live / not-applicable
flowchart LR
D1{"n(s)"}
D1-->|= 0|UNKNOWN[unknown]
D1-->|> 0|D2{"any s : sh=unhealthy"}
D2-->|yes|UNHEALTHY[unhealthy]
D2-->|no|D3{"any s : sh=starting"}
D3-->|yes|STARTING[starting]
D3-->|no|D4{"any s : sh=healthy"}
D4-->|yes|HEALTHY[healthy]
D4-->|no|D5{"any s : sh=unknown"}
D5-->|yes|UNKNOWN
D5-->|no|NONE[none]

The Single Derivation Pipeline (DeriveDeployment)

Section titled “The Single Derivation Pipeline (DeriveDeployment)”

DeriveDeployment runs the same derivation for every set of runtime observations. Identical inputs produce identical results.

The entry point is DeriveDeployment(containers []RawContainer, desiredReplicas map[string]int) -> (DeploymentStatus, StackHealth). It groups the raw observations by service (preserving first-seen order) and runs the chain:

per container -> DeriveContainerStatus + DeriveContainerHealth
per service -> DeriveServiceStatus + DeriveServiceHealth
deployment -> DeriveStackStatus + DeriveStackHealth

Universal input shape: every surface builds a slice of RawContainer from its own probe and hands it to the pipeline, so the transform lives in exactly one place:

type RawContainer struct {
Service string // compose service this container belongs to (grouping key)
RawStatus string // raw docker status: running/exited/created/restarting/paused/dead
RawHealth string // raw docker health: healthy/unhealthy/starting/"", or "unknown" when undeterminable
ExitCode *int // meaningful for exited/dead; nil derives unknown (degraded under always), never complete
RestartPolicy string // service restart policy; normalized on the way in
InFlightConverge bool
}

Surfaces that use the derivation:

SurfaceProbe it builds RawContainer from
Live recorder (propagate.go)DB container rows
Install spinner (compose_progress.go)Runtime project observations
Disk-fallback reader (router.go)label-filtered docker ps

Status derivation consumes raw health; health aggregation consumes DeriveContainerHealth. An unreadable observation stays unknown.

Two variants expose the per-service breakdown and the explicit-service-set case:

  • DeriveDeploymentServices: returns the per-service results (used by the install spinner to fill per-service status + health rows) and the rolled-up deployment status/health.
  • DeriveDeploymentForServices: runs the chain over an explicit, ordered service set so a service with zero observed containers still counts toward total = len(services) (the recorder needs this, or running == total would spuriously fire when a service has lost all its containers). A service absent from desiredReplicas defaults to 1 desired, so a single observed container reads running rather than being judged against zero (which would force stopped).

The node agent owns crash restarts. The runtime’s own restart policy is no; the authored policy lives in labels. At boot the agent waits for the control plane to confirm its lease and assignments before starting containers. A restart requires the replica to remain assigned here, and a single writer also requires a live node lease.

modules/nodeagent/policy.go defines the restart modes and policy labels. The independent watchdog stops single writers after lease expiry even when shcd cannot act. A degraded status alone does not authorise a restart.


SHC exports status metrics to OpenTelemetry:

MetricLabelsDescription
shc_container_statusdeployment, service, container, statusContainer SHC status (gauge=1)
shc_service_statusdeployment, service, statusService status (gauge=1)
shc_deployment_statusdeployment, statusDeployment status (gauge=1)

The gauge value is always 1 when exported, allowing aggregation queries like:

sum by (status) (shc_deployment_status{deployment="myapp"})
MetricLabelsDescription
shc_container_health_statustenant, environment, stack, app, node_id, container_id, container_name, serviceContainer health status (1=healthy, 0=unhealthy, -1=none)
shc_deployment_healthtenant, stack, environmentDeployment health (1=healthy, 0=unhealthy)

flowchart LR
subgraph DOCKER
DE[Docker Events]
DH[Docker Health]
end
subgraph SHC_INGEST
EW[Event Watcher]
HP[Health Poller]
end
subgraph SHC_STATE
CM[ContainerModel]
SM[ServiceModel]
DM[DeploymentModel]
end
DE-->EW
DH-->HP
EW-->CM
HP-->CM
CM-->SM
SM-->DM
SourceEventsFrequency
Docker Eventscreate, start, stop, die, kill, pause, unpause, destroy, oom, health_statusReal-time
Health PollerContainer stats, health statusEvery 60 seconds

When a container event occurs:

  1. Container layer: Update ContainerModel with Docker status/health
  2. Service layer: Recompute ServiceModel.status and ServiceModel.health from containers
  3. Deployment layer: Recompute DeploymentModel.status and DeploymentModel.health from services

Important: Status propagation is triggered when either:

  • Container status changes (e.g., running → exited)
  • Container health changes (e.g., healthy → unhealthy)

This ensures that health check failures properly propagate to deployment status (causing running → degraded).


DerivationInputsOutput
SHC Container Statusds, dh (raw), ec, rpcs
SHC Container Healthdh (raw), dsch
SHC Service Statuscs[], dr, rpss (incl. paused)
SHC Service Healthch[]sh (incl. unknown rung)
SHC Deployment Statusss[], user_action, rtdeployment status (incl. paused)
SHC Deployment Healthsh[]deployment health (incl. unknown rung)
Whole pipeline (DeriveDeployment)[]RawContainer, dr map(DeploymentStatus, StackHealth)

ComponentFileDescription
Container Statusmodules/app/container_status.goDeriveContainerStatus(ds, ec, dh, rp)
Container Health (derived)modules/app/container_status.goDeriveContainerHealth(rawHealth, dockerStatus)
Service Statusmodules/app/service_status.goDeriveServiceStatus(containers, dr, rp)
Service Healthmodules/app/service_status.goDeriveServiceHealth(containers)
Deployment status (derived)modules/app/stack_status.goDeriveStackStatus(services)
Deployment health (derived)modules/app/stack_status.goDeriveStackHealth(services)
Deployment Pipeline (canonical entry)modules/app/deployment_status.goDeriveDeployment / DeriveDeploymentServices / DeriveDeploymentForServices, RawContainer
Deployment status constants (incl. paused)modules/app/types.goDeploymentStatus* consts, IsOperational()
Operational Statesmodules/app/types.goDeploymentStatus.IsOperational()
Status propagation (recorder)modules/app/propagate.gobuilds RawContainer from DB rows, calls the pipeline
Restart policymodules/nodeagent/policy.goAuthored policy labels read by the node agent
Status Metricsmodules/metrics/definitions.goshc_container_status, shc_service_status, shc_deployment_status
Restart Policy Normalizationmodules/app/models.goNormalizeRestartPolicy()

func deriveContainerStatus(dockerStatus, health string, exitCode int, restartPolicy string, converging bool) string {
switch dockerStatus {
case "created":
if converging {
return "starting"
}
return "degraded"
case "restarting":
return "starting"
case "dead":
return "degraded"
case "paused":
return "paused"
case "running":
switch health {
case "starting":
return "starting"
case "unhealthy":
return "degraded"
default:
return "running" // healthy, "", none, or any unrecognized word
}
case "exited":
isRestartNo := restartPolicy == "no" || restartPolicy == "on-failure"
if exitCode == 0 {
if isRestartNo {
return "complete"
}
return "degraded" // daemon exited unexpectedly
}
if restartPolicy == "no" {
return "failed"
}
return "degraded" // will be restarted
}
return "degraded" // catch-all
}
// Derives the CANONICAL ch from the RAW docker health and the RAW status.
// Health is meaningful only while a container is live, so it is a function
// of BOTH inputs, not the raw health alone.
func deriveContainerHealth(rawHealth, dockerStatus string) string {
if rawHealth == "unknown" {
return "unknown"
}
if dockerStatus != "running" {
return "none" // not live -> no active check -> not-applicable
}
switch rawHealth {
case "unhealthy":
return "unhealthy"
case "starting":
return "starting"
case "healthy", "", "none":
return "healthy" // live + (check passing OR no check) = healthy
default:
return "unknown" // a live container with a word we don't recognize
}
}
func deriveServiceStatus(containers []Container, desiredReplicas int, restartPolicy string) string {
if desiredReplicas == 0 {
return "stopped"
}
var running, paused, restarting, starting, complete, failed int
for _, c := range containers {
switch c.Status {
case "running":
running++
case "paused":
paused++
case "starting":
starting++
case "complete":
complete++
case "failed":
failed++
}
// "restarting" is inferred from the raw docker status by the
// recorder; the container tier maps it to "starting" today, so
// it is rarely a distinct count here.
}
total := len(containers)
// Whole-service pause (e.g. a backup quiesce), checked before the
// running rule so an all-paused service does not read "running".
if total > 0 && paused == total {
return "paused"
}
// Paused counts as "alive enough"; a partial pause lands here as
// "running" rather than flapping the status.
if running+paused >= desiredReplicas {
return "running"
}
if restarting > 0 {
return "restarting"
}
if starting > 0 || total == 0 {
return "starting"
}
if restartPolicy == "no" || restartPolicy == "on-failure" {
if complete == total {
return "complete"
}
if failed == total {
return "failed"
}
}
return "degraded" // catch-all (incl. daemon mode)
}
func deriveServiceHealth(containers []Container) string {
if len(containers) == 0 {
return "unknown"
}
// Container health here is the DERIVED ch, not the raw docker health.
hasUnhealthy, hasStarting, hasHealthy, hasUnknown := false, false, false, false
for _, c := range containers {
switch c.Health {
case "unhealthy":
hasUnhealthy = true
case "starting":
hasStarting = true
case "healthy":
hasHealthy = true
case "unknown":
hasUnknown = true
}
// "none" (non-live containers) contributes nothing.
}
if hasUnhealthy {
return "unhealthy"
}
if hasStarting {
return "starting"
}
if hasHealthy {
return "healthy"
}
if hasUnknown {
return "unknown"
}
return "none" // all containers non-live / not-applicable
}

Operational states (queued, created, installing, starting, stopping, updating, removing) block derived updates; callers check status.IsOperational() and skip the derivation entirely. The pure derivation over the per-service statuses is count-based:

func deriveDeploymentStatus(services []Service) string {
if len(services) == 0 {
return "stopped"
}
var running, starting, complete, failed, stopped, degraded, paused int
total := len(services)
for _, s := range services {
switch s.Status {
case "running":
running++
case "starting":
starting++
case "complete":
complete++
case "failed":
failed++
case "stopped":
stopped++
case "degraded":
degraded++
case "paused":
paused++
}
}
switch {
case paused == total:
return "paused" // whole deployment paused (e.g. backup quiesce)
case running == total:
return "running"
case starting > 0:
return "starting"
case complete == total:
return "complete"
case failed == total:
return "failed"
case stopped == total:
return "stopped"
case running+complete == total && running > 0:
return "running"
case running+paused+complete == total && (running > 0 || paused > 0):
return "running" // partial pause w/ no bad state -> alive
case degraded > 0:
return "degraded"
case running > 0 || failed > 0 || paused > 0:
return "degraded" // paused mixed with a bad state -> degraded
case complete > 0:
return "complete"
}
return "stopped"
}

Status and health are independent concepts:

  • Status: Is the container/service/deployment running?
  • Health: Is the application inside working correctly?

A service can be running but unhealthy (app responding but health check failing).

Deployment status has two categories:

  • Operational: Set by user commands, blocks derived updates during operations
  • Derived: Computed from service states when no operation is in progress

This prevents status flickering during deployments.

The restart policy tells SHC what the user expects:

  • no / on-failure: Job mode; exit is expected
  • always: Daemon mode; should always be running

The same Docker status (exited) produces different SHC statuses based on intent.

SHC selects the node runtime for execution. The node agent uses assignments and lease state to decide whether a container may start or restart.

compose up -d is idempotent; running it multiple times is safe. Combined with exponential backoff (60s base doubling to a 1800s cap, applied only after a failed attempt), this prevents rapid repeated actions.


Podman container states are nearly identical to Docker:

StateDockerPodmanDifference
createdYesYesSame
runningYesYesSame
pausedYesYesSame
exitedYesYesSame
restartingYesYesSame
deadYesNoPodman doesn’t have this
configuredNoYesPodman-only (config exists, not initialized)
removingYesYesSame

SHC treats Podman containers the same as Docker containers. The configured state maps to starting in SHC, and if dead is not present, SHC won’t see it (containers go directly to exited).


For context, here’s how Kubernetes handles similar concepts:

PhaseDescription
PendingScheduled but not yet running
RunningAt least one container running
SucceededAll containers exited with code 0
FailedAt least one container exited with non-zero
UnknownCannot determine state (node communication issue)

Deployment Status (equivalent to deployment)

Section titled “Deployment Status (equivalent to deployment)”
status:
replicas: 3 # Desired
readyReplicas: 2 # Actually ready
availableReplicas: 2 # Passed readiness probe
unavailableReplicas: 1
conditions:
- type: Available
status: "True" # At least minAvailable replicas ready
- type: Progressing
status: "True" # Rollout in progress or complete
  • Declarative: You specify desired state, K8s reconciles continuously
  • Self-healing: Deleted pods are immediately recreated
  • Pause not supported: No concept of pausing a pod (you scale to 0 instead)
  • Rich status: Conditions, replica counts, rollout status all available
  • Probe-based health: Liveness and readiness probes determine health
KubernetesSHC Equivalent
Pod RunningContainer running
Pod SucceededContainer complete
Pod FailedContainer failed
Deployment Available=TrueDeployment running
Deployment Progressing=TrueDeployment updating or starting
Deployment replicas mismatchDeployment degraded
Readiness probe passingContainer health healthy
Liveness probe failingContainer killed, status becomes degraded