Skip to content
SHC Docs

Subsystem lifecycle management

Layer 2. Core. Core

The SubsystemManager is the central orchestrator for every background process inside the SHC server. It provides one place to register, start, and stop subsystems with explicit ordering, dependency declarations, rollback on startup failure, and signal- driven shutdown.

The single-writer watchdog runs outside shcd so it can stop labelled containers after the node lease expires even when shcd is hung. See Runtime support.

  1. Unified lifecycle: one place to understand all background things.
  2. Explicit ordering: Priority + Deps enforce startup order; reverse for shutdown.
  3. Fail-fast: a required subsystem failing at start triggers rollback and crashes the server (systemd restarts).
  4. No in-place restart (by design): per B8 there is no runtime restart of an individual subsystem; the recovery unit is the whole process (systemd restarts it cleanly). Leader-only work is gated in place via LeaderAware.OnLeader, not by stopping/starting the subsystem.
  5. Composition root, not service locator: the daemon’s module services are owned by a single composition root, daemon.Services (daemon/composition.go), built in dependency order before lifecycle.Wire and injected into consumers (HTTP/CLI builders via registry.Deps.Services; subsystems via the lifecycle Resolver).

The registration list lives in subsystems/register.go; each subsystem’s Priority() is declared in its own impl/*.go constructor (the numbers below are read from those).

Selected subsystems register under the Lifecycle orchestrator (subsystems/lifecycle):

SubsystemPrioritysvcRole
Bootstrap-50first-run setup
Vault-30vault unlock
ContainerRuntime-20svcdocker / runtime detection
Collectors-10svcOTel exporters + tick cron
RqliteEngine1svcrqlited supervisor + pool swap (no-op without the engine marker)
SSEBroker1svc
ConfigLoader1load DB configuration
EventBus2svc
StackReconcile4reconcile a deployment’s desired state (subsystems/impl/stack_reconcile.go)
Migrations5atlas migrations against the post-swap pool
EntClient6svcopens ent.Client against the post-swap pool
NodeSelfRegister6ModeAny: leader upserts its own row; follower fills worker fields
BackupMount6svcFUSE backup-mount manager
CAGenesis7cluster CA and app-root CA reconciliation
Queue8svc
DockerEvents9svc
ContainerRecorderNode11svc
UnixSocket20svc/run/shc/shc.sock listener
HTTP30svcTCP API listener
InternalAPI40svcmTLS internal listener (cluster mode)
InternalCertRotation45svcper-node :4003 x509 leaf renew (ModeAny)
LeaderHub50svcpolls raft leadership, fans out OnLeader
ContainerReconciler51svc
VolumeSizeSweep56svconce per leadership gain, records missing sizes of live block volumes
IngressMaterialize65svcModeAny: self-gates on Traefik-host placement
ProxyReconcile66svcleader-only L4 proxy-plane reconcile
BackupScheduler70svcleader-only cron-driven backups
OperationRegistry70svcSSE-streamed long-running op store
ScheduleDispatcher71svc
CronRunner72svcowns every registry.CronJob (deps queue, leader_hub)

VolumeSizeSweep runs once on each leadership gain, including startup as leader. It reads all tenants, skips kept or releasing volumes, and asks providers sequentially for missing sizes. Leadership loss cancels the pass; the next gain retries rows that still have no size. Every pass emits one X200311VOLSWP summary, including zero counts. It has no periodic timer.

Legend: svc = service-style subsystem (owns a goroutine until Stop). The Priority column is the Priority() each subsystem declares, and the lifecycle resolver topologically sorts by priority + Deps() at Wire time, so registration order is incidental.

TypeDescriptionExamples
One-shotRuns once during Start, then markStarted() and exitsBootstrap, Migrations, NodeSelfRegister, ConfigLoader, StackReconcile, CAGenesis
ServiceLong-running goroutine driven by ctxSSEBroker, EventBus, Queue, DockerEvents, LeaderHub, Collectors, CronRunner
ListenerHTTP / WS / UDS listenersUnixSocket, HTTP, InternalAPI
Leader-onlyLeaderOnly: true. Stays Running on every node. Only its leader-gated work toggles via OnLeader. (Modes separately gates a subsystem off entirely on the wrong cluster role.)BackupScheduler, ProxyReconcile
  • -50 to -10: bootstrap, vault, runtime, collectors
  • 1: rqlite engine (no-op without the engine marker; the pool swap to the raft engine must land before migrations)
  • 1-11: config + event infrastructure + reconcile + migrations + ent client + node self-register + backup mount + cluster CA + queue
    • docker events + container recorder
  • 20-29: local API (Unix socket)
  • 30-49: network listeners (HTTP, internal API) + per-node cert rotation
  • 50-59: leader detection + container reconciliation
  • 60-72: cluster cert rotation, ingress materialize, proxy reconcile, backup scheduler, operation registry, schedule dispatcher, cron runner

Voter membership is not a subsystem. The leader cron voter_suffrage runs every minute and adjusts voter roles one member at a time. A down member keeps its vote until an operator removes it. Source: CE modules/cluster/register.go and modules/cluster/suffrage.go.

type Subsystem interface {
Name() string
Modes() Mode // bitmask of run modes (ModeAny when always-on)
Deps() []string // names that must be running first
Priority() int
LeaderOnly() bool
Start(ctx context.Context, deps any) error // Idempotent. deps is built by the Registration's DepsBuilder.
Stop(ctx context.Context) error // Idempotent.
HealthCheck(ctx context.Context) error // nil = healthy; non-nil = Degraded.
}

LeaderAware is an optional second interface: a subsystem that implements OnLeader(ctx, isLeader) is driven by the leader hub on every transition (and once at Wire time with the initial state).

Each subsystem is installed through a lifecycle.Registration (subsystems/lifecycle/registry.go) carrying:

  • Priority: startup order
  • Deps: list of subsystem names that must be running first
  • Modes: bitmask of run modes (ModeAny for always-on; panics if zero)
  • LeaderOnly: gates the subsystem’s leader-work behind raft leadership
  • StartIf: optional func(*Env) bool evaluated against the Env at Wire time; if it returns false, the subsystem is skipped
  • DepsBuilder: func(*Resolver) any that constructs the typed deps value handed to Start (this is where a subsystem pulls Resolver.Services, the composition root, and the already-started subsystems it depends on)
  1. cmd/shcd/main.go delegates to daemon.Run (daemon/daemon.go), which loads config, initialises logging + OTel, opens the database pool, and calls subsystems.RegisterAll().
  2. daemon.Run then builds the composition root with newServices() and populates it via wireModuleServices + wireExtraModuleServices
    • wireModuleSeams (all in daemon/services*.go, ce module), passing a moduleServiceDeps{pool, cfg, logger, paths, services}. Each per-module setter stores its service on Services. This runs BEFORE lifecycle.Wire, because the unix_socket subsystem walks registry.Routers() at Wire time and each router closure captures its service then.
  3. lifecycle.Wire(ctx, WireOptions{..., Services: services}) returns the *Lifecycle after every subsystem is registered; the composition root rides through on WireOptions.Services and lands on the resolver as Resolver.Services.
  4. Subsystems carry priority + Deps(); the resolver topologically sorts them at Wire-time.
  5. SIGINT/SIGTERM are bound in cmd/shcd/main.go via signal.NotifyContext; the resulting cancellable ctx is threaded into daemon.Run. lifecycle.Wire then starts subsystems in priority groups (the resolver enforces Deps()); the first group member to fail cancels its siblings, rolls everything already started back in reverse-priority order, and returns the error.
  6. lc.Wait(ctx) blocks until that ctx cancels (the SIGTERM/SIGINT path) or a FatalHealthError surfaces.
  1. ctx cancellation makes lc.Wait(ctx) return; daemon.Run calls lc.Stop(stopCtx) with a 60 s bounded context.
  2. Subsystems stopped in reverse priority order.
  3. If a subsystem doesn’t yield within the stop deadline it’s considered stopped and the orchestrator moves on.
  4. daemon.Run returns; OTel + DB pool shutdown run in defer blocks.

Every subsystem is required; there is no optional/continue-on-error start path. Any Start (or DepsBuilder) failure marks the subsystem StateErrored, cancels its priority-group siblings, rolls back every already-started subsystem in reverse-priority order, and returns the wrapped coded error from daemon.Run → the process exits → systemd restarts it.

Subsystems are gated on cluster state through three independent knobs: Modes (off entirely on the wrong cluster role), LeaderOnly (stays Running but quiesces leader-work off the leader), and StartIf (an Env predicate evaluated at Wire time):

SubsystemActive when
RqliteEnginealways-on (ModeAny, a no-op without the {state}/engine.json marker; supervises the external rqlited process on promoted nodes)
NodeSelfRegisterModeAny: the leader self-registers its own row from on-disk bootstrap files; a follower fills in the worker-side fields
InternalAPIalways registered; StartIf gates on Env.InternalAPIOn (NOT HasCertsOnDisk. Certs may post-date vault unlock); binds the node mesh address with mTLS

There is deliberately NO global service-locator for the lifecycle (B8 bans the GetSubsystemManager() pattern). The daemon holds the *Lifecycle handle returned by lifecycle.Wire and may consult it:

lc, err := lifecycle.Wire(ctx, opts)
// ...
state := lc.State("rqlite_engine") // one subsystem's State
all := lc.States() // name -> State snapshot
names := lc.Running() // currently-running names
sub, ok := lc.GetSubsystem("event_bus")

Cross-subsystem dependencies flow through the typed DepsBuilder (lifecycle.Resolver) at Start time.

Module services (as distinct from subsystems) are owned by the composition root daemon.Services (daemon/composition.go). Services are built in dependency order and injected into consumers.

  • HTTP / CLI builders read registry.Deps.Services (type-erased to any to avoid a registry → daemon import cycle; a builder type-asserts the small accessor interface it needs, e.g. d.Services.(interface{ UserService() *user.Service })).
  • Subsystems read Resolver.Services inside their DepsBuilder (also any, to keep lifecycle free of a daemon import) and thread it into Start.

Two publishing shapes coexist on the root:

  • Daemon-built leaves (W1a/W3): services built inside a wireX function in the daemon package (user, schedule, audit, app, clone, …) are stored via the UNEXPORTED setters (setUser, setApp, …) called directly.
  • Subsystem-published (W1b): services constructed inside a lifecycle subsystem in subsystems/impl (events from sse_broker/event_bus, docker from container_runtime, task from queue, config from config_loader, vault, cluster, backup, health, metrics, leader) reach the root through EXPORTED Set<X>Service methods. impl cannot name *daemon.Services (impl → daemon would cycle), so each subsystem declares the small publisher interface it needs and the Set<X>Service method satisfies it (Dependency Inversion).

The lifecycle verbs are Wire and Stop. A subsystem signalling FatalHealthError makes Wait return and shuts down the daemon; recovery is a process restart. Sources: CE subsystems/lifecycle/lifecycle.go and daemon/daemon.go.

  • Leadership changes: lc.LeaderTransition(ctx, isLeader) broadcasts to every LeaderAware subsystem’s OnLeader, which starts/cancels internal goroutines in place (see container_reconciler).
  • Config changes: subsystems watch *config.Config.OnChange or re-read config per tick; the config_loader reloads the live tree when DB-backed rows change.
  • Self-healing: health checks flip Running <-> Degraded without goroutine bounces; terminal failures are fatal on purpose.
StateDescription
StateRegisteredDeclared in init(), not yet started
StateStartingStart() in flight
StateRunningStart() returned nil; HealthCheck green
StateDegradedHealthCheck failing, goroutines still owned
StateStoppingStop() in flight
StateStoppedStop() returned cleanly
StateErroredTerminal: Start() failed or FatalHealthError

Unless WireOptions.SkipHealth is set (test mode), lifecycle.Wire spawns one healthLoop per subsystem that ticks HealthCheck on a cadence: a non-nil result flips Running → Degraded (and back on recovery) without bouncing goroutines, and a FatalHealthError flips Errored, closes the fatal channel, and makes Wait return so the daemon shuts down (fail-fast); systemd restarts it.

The leader_hub subsystem (subsystems/impl/leader_hub.go) polls the raft leader id against {state}/node.id at config.leader_poll_interval, and on each become-leader / lose-leader edge drives the transition through Lifecycle.LeaderTransition, which fans OnLeader(ctx, isLeader) out to every subsystem implementing the lifecycle.LeaderAware interface. In single-node mode the hub still starts but IsLeader returns true unconditionally.

flowchart TB
hub["leader_hub<br/>polls raft leadership"] -->|"become/lose-leader edge"| lt["Lifecycle.LeaderTransition"]
lt --> walk["lifecycle walks every LeaderAware subsystem"]
walk -->|"OnLeader(ctx, isLeader)"| gated["BackupScheduler / ProxyReconcile<br/>(LeaderOnly)"]
walk -->|"OnLeader(ctx, isLeader)"| aware["ContainerReconciler, ...<br/>(any LeaderAware subsystem)"]

To make a subsystem leader-aware, set LeaderOnly: true on its Registration and implement LeaderAware.OnLeader:

func (s *MySubsystem) OnLeader(ctx context.Context, isLeader bool) error {
if isLeader {
return s.startLeaderWork(ctx)
}
return s.stopLeaderWork()
}

OnLeader is also called once at Wire time with the initial leadership state, so a subsystem that started before the hub settled catches up. The subsystem stays Running on every node; only its leader-gated internal work toggles.

subsystems/
├── lifecycle/ # Subsystem protocol, Lifecycle orchestrator, State enum
│ ├── lifecycle.go # Wire / Stop / LeaderTransition / startGroup / health
│ ├── registry.go # Registration, Resolver (incl. Services), Register
│ ├── subsystem.go # Subsystem + LeaderAware interfaces, baseSubsystem
│ ├── topo.go # priority + Deps topological sort
│ ├── interfaces.go
│ ├── bridge.go
│ ├── errors.go
│ └── types.go # Mode, State enum
├── impl/ # Concrete subsystems
│ ├── bootstrap.go # Bootstrap
│ ├── vault.go # Vault
│ ├── container_runtime.go # ContainerRuntime
│ ├── collectors.go # Collectors (tick cron + metric collectors)
│ ├── rqlite_engine.go # RqliteEngine (rqlited supervisor + pool swap)
│ ├── sse_broker.go # SSEBroker
│ ├── config_loader.go # ConfigLoader
│ ├── event_bus.go # EventBus
│ ├── stack_reconcile.go # StackReconcile
│ ├── migrations.go # Migrations (atlas runner)
│ ├── ent_client.go # EntClient (opens ent.Client on post-swap pool)
│ ├── node_self_register.go # NodeSelfRegister
│ ├── backup_mount.go # BackupMount
│ ├── ca_genesis.go # CAGenesis
│ ├── queue.go # Queue (task.Worker)
│ ├── docker_events.go # DockerEvents (event collector)
│ ├── container_recorder_node.go # ContainerRecorderNode
│ ├── unix_socket.go # UnixSocket listener
│ ├── http.go # HTTP listener
│ ├── internal_api.go # InternalAPI (mTLS)
│ ├── internal_cert_rotation.go # InternalCertRotation (per-node :4003 leaf)
│ ├── leader_hub.go # LeaderHub
│ ├── container_reconciler.go # ContainerReconciler (LeaderAware)
│ ├── ingress_materialize.go # IngressMaterialize
│ ├── proxy_reconcile.go # ProxyReconcile
│ ├── backup_scheduler.go # BackupScheduler
│ ├── schedule_dispatcher.go # ScheduleDispatcher
│ ├── cron_runner.go # CronRunner (owns every registry.CronJob)
│ ├── operation_registry.go # OperationRegistry
│ └── *_publisher.go # consumer-defined publisher interfaces → daemon.Services
└── register.go # RegisterAll: installs every subsystem above
daemon/
├── composition.go # Services (the DI composition root) + per-service accessors/setters
├── services.go # wireModuleServices + moduleServiceDeps (W1a leaves)
├── services_extra.go # wireExtraModuleServices
├── services_seams*.go # wireModuleSeams + the cross-module resolver seams
└── daemon.go # Run: RegisterAll → newServices → wireX → lifecycle.Wire
cmd/shcd/main.go # entry point: cobra, then daemon.Run
  1. Register in priority order in code for readability.
  2. Make Start() and Stop() idempotent (embed baseSubsystem).
  3. Long-lived loops must NOT derive their context from the Start ctx; startGroup cancels the group’s start context once the priority group is up. Root loops in context.Background() (or the OnLeader ctx) and let Stop own the cancel.
  4. Use StartIf (a func(*Env) bool on the Registration) for runtime-state-dependent subsystems (certs on disk, internal API enabled, …) and Modes for leader/follower gating.
  5. Periodic work can live either as a subsystem loop or as a registry.CronJob. The cron_runner subsystem owns every registered CronJob: leader-gated jobs (rpc-crons always; task-crons when LeaderOnly) tick only while this node is leader, while node-local jobs tick on every node for the whole Start..Stop window.
  6. Leader-gated work: set LeaderOnly: true and implement LeaderAware.OnLeader (the leader_hub subsystem drives the transitions); the subsystem stays Running on every node; only its internal work toggles.