Skip to content
SHC Docs

Metrics module

Layer 4. Primitives. Primitives Readings have no owned record.

Source: modules/metrics/ and subsystems/impl/collectors.go.

The metrics module wires SHC’s OpenTelemetry metrics pipeline: counters, gauges, and histograms for HTTP requests, database queries, task queue depth, docker operations, per-stack quota usage, and cluster-wide rollups. Metrics export to an OTLP collector (OpenObserve by default; see collector in default.yaml).

The pipeline has three parts:

  1. Stats cache (stats_cache.go): in-memory per-container + per-node readings the docker subsystem pushes on every tick. Hot reads serve Status, Inspect, and HUD render paths under sync.RWMutex (≈37ns/op for single-key reads; see BenchmarkGetContainer).
  2. Service (service.go): the emit surface every other module calls via metrics.GetRuntimeService(). Wraps one otel.Meter and applies cardinality caps + diagnostic-cardinality dropping per B9 fix #3.
  3. Tick cron + collectors (modules/metrics/collectors/ and the collectors subsystem at subsystems/impl/collectors.go): periodic emitters that scan the cache + DB and call Service.RecordGauge / RecordCounter. The tick cron runs every 2s by default (timing.tick_interval).

Registered in priority order at Collectors.Start:

CollectorLeader-onlySource
collect_node_ticknowrites the stats cache from the docker daemon
nodenoper-node CPU/MEM/disk gauges
containernoper-container gauges (one entry per running container)
subsystem_statenolifecycle state per subsystem
quotayesper-(tenant, env, stack) quota readings
aggregateyesfour-tier rollup (service / stack / env / tenant)
schedulingnocapacity vs scheduled per node

Leader-only collectors run only on the rqlite leader so followers don’t double-emit cluster-wide signals.

shc_stack_disk_usage_bytes (and its environment and tenant siblings) publish only when something measured the bytes. A stack’s disk is the bytes its block volumes hold, as the nodes holding them last measured them (storage.StackVolumeBytes, FZ-7801; see storage). A stack with no measured volume publishes no gauge and shc.metrics.stack sends disk_bytes: null for it; data a deployment keeps on the node disk is quota’s to measure, not this gauge’s.

It used to publish a hard 0 instead, from a rollup field nothing ever assigned. Measured on the corp estate 2026-09-07: all 91 rows of the console’s estate read carried disk_bytes: 0, the springboard status strip read “Disk 0 B”, and the disk glyph could never change colour; on an estate holding hundreds of GB.

A consumer must therefore treat disk_bytes: null as “no reading” and draw nothing, not ?? 0. A measured zero still travels as 0 and still prints: absence and idle are different facts (SHC-566).

Per B9-observability.md Phase 3, every known non-default tenant gets its own scope-named Meter alongside the base set:

mp := daemonMeterProvider
baseMeter := mp.Meter("shc")
acmeMeter := mp.Meter("shc.tenant.acme")
mangoMeter := mp.Meter("shc.tenant.mango")

Each per-tenant Meter backs a separate *metrics.Service and a tenant-filtered variant of the Container / Quota / Aggregate collectors (NewContainerCollectorScoped etc.). The scoped collector iterates the same source data but only emits rows whose TenantName matches; the emit goes through the tenant’s Meter so the OTLP instrumentation_scope.name attribute carries the tenant identifier.

Downstream OTLP receivers (Mimir, Tempo, OpenObserve, etc.) can filter metrics by scope.name directly:

shc_container_running{scope_name="shc.tenant.acme"}

…instead of relying on the tenant label, which is still emitted but joins less efficiently in large clusters. Single-tenant installs see no per-tenant scope, only the base shc Meter.

The collectors subsystem re-enumerates tenants every 30 ticks (≈60s by default). A new tenant created at runtime starts emitting under its scope on the next refresh; the per-tenant metrics.Service allocates its own instruments on demand.

Failures of per-tenant service construction (cardinality cap, etc.) log X503056COLPRT and skip that tenant for the cycle; the base collectors keep emitting normally.

Per-tenant Resource attributes (not just scope name) are NOT differentiated. OTel SDK ties Resource to MeterProvider, and the daemon initializes a single MeterProvider at boot. Per-tenant Resource would require a per-tenant CollectorConfig surface that doesn’t exist yet; it’s a separate workstream documented in the B9 follow-up.

modules/metrics/
├── service.go # emit surface (RecordCounter/Gauge/Histogram)
├── stats_cache.go # in-memory per-container + per-node readings
├── types.go # ContainerCacheEntry, NodeStats
├── collectors/
│ ├── container.go # per-container gauges + tenant filter
│ ├── node.go # per-node gauges
│ ├── quota.go # per-scope quota gauges + tenant filter
│ ├── aggregate.go # four-tier rollup + tenant filter
│ ├── subsystem_state.go
│ ├── scheduling.go
│ └── system_tick_source.go
└── ...

The collectors subsystem at subsystems/impl/collectors.go owns the tick cron + the per-tenant service wiring.