Observability
SHC emits four signals operators can use to see what’s happening inside a running cluster:
- HTTP health / status endpoints: quick liveness checks
- Structured logs: every event the daemon emits
- Metrics: internal gauges/counters/histograms, including node-exporter/cAdvisor-compatible series for containers
- Traces: request-scoped spans across the API surface
All four are push-based (OTLP-HTTP) to an observability backend rather than scraped. The destination is configured once (or derived automatically from the system OpenObserve app), and SHC streams to it. See OTLP export configuration below.
Health endpoints
Section titled “Health endpoints”Two health surfaces, on two different listeners; pick by who is asking:
GET /api/method/shc.status.get (public, through Traefik on 443)
Section titled “GET /api/method/shc.status.get (public, through Traefik on 443)”The cluster status method on the public API, which off the node is
Traefik’s shc-api route on the estate’s host over https.
Deliberately exempt from auth and the quorum-required middleware: it
must answer even when the control plane is degraded. Returns the
standard { "message": ... } envelope:
{ "message": { "nodes": [ { "node": "shc-node-1", "state": "healthy", "cpu_percent": 12.4 } ], "deployments": [ { "stack": "keycloak", "state": "running" } ], "last_poll": "2026-07-19T00:00:00Z" }}Source:
modules/health/router.go
(GET and POST serve the same projection).
Being unauthenticated is by design, so treat it as public surface: restrict it at Traefik or your load balancer rather than expecting auth.
node.health (peers only, internal RPC on mTLS port 4003)
Section titled “node.health (peers only, internal RPC on mTLS port 4003)”The node-liveness probe peer daemons poll. It is an internal RPC
function (POST /internal/rpc/node.health on the internal listener,
:4003), mTLS-only, so it is not reachable without a
cluster-issued client certificate. Reports the node’s local state
(healthy / degraded / unhealthy from disk + memory thresholds),
uptime, leader flag, and system metrics. Source:
modules/health/internal_router.go.
Using the status method from a load balancer
Section titled “Using the status method from a load balancer”Point the check at Traefik’s https route on the estate’s host. shcd’s
own :8000 answers only the node, its system ingress and the mesh, and
the internal :4003 listener rejects anything without a cluster client
cert:
HealthCheck: Protocol: HTTPS Host: <the estate's host> Path: /api/method/shc.status.get Port: 443 Interval: 10s HealthyThreshold: 2 UnhealthyThreshold: 3 Timeout: 5sSHC does not use bare stdlib log; it emits structured events
through the log module
(core/log/log.go), which fans
records out to the console/journal AND (when OTLP export is active)
to the OTel log bridge.
Every log record is a structured object with:
timestampseverity:DEBUG,INFO,WARN,ERROR(there is no TRACE or FATAL tier)body: human messageattributes: dict of contextual key/value pairstrace_id,span_id: if emitted inside a traced requestresource:{ tenant, stack, environment, service, container, node_id }as available
Viewing logs
Section titled “Viewing logs”Three ways in:
| Tool | Use for |
|---|---|
shc events --follow | Live-stream the daemon’s event log |
shc events --audit | View the audit trail (subset of log events; see security) |
| Remote viewer | If OTLP export is active (see OTLP export configuration), daemon log records (INFO and above) are also pushed there as OTLP logs (default target: the system OpenObserve; see the openobserve app) |
Error codes
Section titled “Error codes”Every error / warning event carries a code of the form
X<HTTP><SEQ><MNEMONIC> (e.g. X500908HKFAIL). The code is stable
across releases and documented in the
error reference; if you see a code in
a log or user-facing response, that’s the canonical place to look up
what it means.
Metrics
Section titled “Metrics”In-process metric registry
Section titled “In-process metric registry”Every SHC module declares the metrics it emits via the metrics
helper
(core/otel/otel.go). Three primitives:
- Counter: monotonically increasing integer
- Gauge: point-in-time value
- Histogram: bucketed distribution;
HistogramTimertimes operations via a context manager
Node / container series (node-exporter & cAdvisor compatible)
Section titled “Node / container series (node-exporter & cAdvisor compatible)”In addition to SHC-internal metrics, each node collects host and
container stats and emits them under names that match node-exporter
1.x and cAdvisor exactly
(modules/metrics/definitions.go, ce module).
That means standard Grafana dashboards work without translation:
| Metric | Source | Compatible with |
|---|---|---|
node_cpu_seconds_total{cpu,mode} | Per-node | node-exporter |
node_memory_MemTotal_bytes, node_memory_MemFree_bytes, node_memory_MemAvailable_bytes, node_memory_Buffers_bytes, … | Per-node | node-exporter |
container_cpu_usage_seconds_total | Per-container | cAdvisor |
container_memory_usage_bytes, container_spec_memory_limit_bytes | Per-container | cAdvisor |
container_last_seen | Per-container | cAdvisor |
Full list: see
prom_definitions.go.
Tested against Grafana dashboard 1860 (Node Exporter Full) and dashboard 893 (Docker Container).
Where they go
Section titled “Where they go”SHC does not currently expose a /metrics scrape endpoint. All
metrics are pushed (OTLP-HTTP) to the configured export destination;
see OTLP export configuration. The
/metrics path is reserved (exempt from auth in middleware) but no
handler is installed.
If you need a scrape-style workflow, point the OTLP export at an OpenTelemetry Collector with a Prometheus exporter, or run a backend that accepts OTLP directly (OpenObserve, Mimir).
Per-tenant instrumentation_scope.name
Section titled “Per-tenant instrumentation_scope.name”Beyond the tenant label every metric already carries, the collector
subsystem emits per-tenant metrics under a distinct OTel
instrumentation scope so dashboards can filter at the scope level
rather than scanning labels:
| Tenant | OTLP scope.name |
|---|---|
baseline / default / node-wide | shc |
acme | shc.tenant.acme |
mango | shc.tenant.mango |
Both representations are emitted simultaneously: the base set with
the tenant label, and the tenant-scoped set without it. The scope
attribute is materialised by OTel SDK when a different Meter() name
is requested from the same MeterProvider, so single-tenant installs
incur no overhead (no extra scope is created).
The fan-out covers three collectors: container, quota, and
aggregate. Node-wide collectors (node, subsystem_state,
scheduling, collect_node_tick) stay on the base scope; they
have no tenant axis to differentiate.
Operators using Mimir / Tempo / OpenObserve can write queries like:
sum by (stack) (shc_container_running{scope_name="shc.tenant.acme"})…to filter to one tenant without scanning the label cardinality of the whole cluster.
The tenant set refreshes every 30 ticks (≈60s default). A newly- created tenant starts emitting under its scope on the next refresh.
For implementation details and limitations (per-tenant Resource is NOT differentiated; that’s a separate workstream), see architecture/modules/metrics.md.
Traces
Section titled “Traces”Every API request opens a span. Child spans wrap:
- Database queries
- Docker client calls
- Inter-node API calls (over mTLS)
- Hook / integration-phase job execution
Spans propagate trace_id / span_id across internal API
boundaries via request headers so a single request can be traced
across the leader and a worker node.
Traces are pushed to the same OTLP destination as logs and metrics.
Source:
core/otel/otel.go (Tracer, Span,
CurrentSpan).
OTLP export configuration
Section titled “OTLP export configuration”All export is standard OTLP-HTTP from the OpenTelemetry SDK:
there is no bespoke collector process and no custom record format.
Each signal POSTs to the standard per-signal path under the configured
base: <base>/v1/traces, <base>/v1/metrics, <base>/v1/logs.
Signals shipped today:
| Signal | Contents |
|---|---|
| Traces | Daemon API request spans + child spans (DB, docker, inter-node calls) |
| Metrics | Daemon-internal counters/gauges/histograms, plus the node-exporter / cAdvisor-compatible node & container series above |
| Logs | The daemon’s structured log records (INFO and above, secrets redacted) via the slog → OTel bridge |
Not yet shipped: application/container log shipping (your app
stacks’ stdout/stderr). Container metrics are shipped; use
shc logs -s <stack> / docker logs for app logs.
shc logs --where 'ordinal=1' -s <stack> prints every replica that
matches. When more than one matches, each line is prefixed with the
replica name, the same way a multi-replica log already is. Several
matches are all shown. When nothing matches, the refusal lists each
replica’s labels. --where together with a replica name such as
web-r1 is refused.
The managed default (zero config)
Section titled “The managed default (zero config)”When the system OpenObserve app is deployed (it is part of the
default module set) and the cluster has a public host
(shc init --ingress-host), the daemon auto-wires itself to it: endpoint
https://<SHC_HOST>, path base /otel/api/default. Each node’s
exporter reads the openobserve root credential from that node’s own
vault on every export, so ingest is authenticated on every node as soon
as its vault unlocks, with no restart. While the vault is locked the
exporter sends nothing, and shc doctor reports it under
app.telemetry-export. A leader reconcile pass also persists the
managed provider, writing a boot-readable drop-in
(/etc/shc/config.d/otel-managed.yaml, mode 0600, since it holds the
Basic-auth credential) plus the global config rows, and emits
X200506OTLPST. A daemon that found no managed destination at startup,
because OpenObserve was not set up yet, built no exporter and starts
exporting at its next restart. No configuration required.
Without a public host. On a cluster with no SHC_HOST there is no
address every node can reach OpenObserve by, so each node’s daemon hands
its exports to the node that runs OpenObserve over the cluster’s
internal RPC, and that node forwards them to its local OpenObserve with
the credential from its own vault. The credential never leaves that
node. The relay follows OpenObserve when it is installed later or moves
to another node, with no restart. The receiving node queues at most 256
exports or 64 MiB, drops the oldest past that, and gives up an export
it could not forward for five minutes; the queue reports pressure as
otel.relay (see the metrics reference).
shc doctor reports a node whose relayed exports are not getting
through under app.telemetry-export, with the reason. The per-node
collector is a container and cannot use the relay: on such a cluster
only the node that runs OpenObserve ships container metrics.
Explicit providers
Section titled “Explicit providers”An explicit provider always overrides the managed default. Set it
under otel: in your config (see
config-files reference):
otel: provider: obs providers: obs: endpoint: https://observe.example.com url_path_base: "" # empty = standard /v1/* paths insecure: false # true = plaintext http:// (dev only) headers: Authorization: ref+vault://otel_tokenSource of truth:
default.yaml (the otel: block),
core/otel/otel.go (exporter assembly).
Any OTLP-HTTP-compatible backend works (OpenObserve, Grafana Cloud,
Honeycomb, an OpenTelemetry Collector, …). TLS verification uses the
union of the host’s system roots and the cluster’s internal global
root, so both public SaaS endpoints and the SHC-internal
https://<SHC_HOST>/otel route verify without any insecure toggle.
Degraded-mode behaviour
Section titled “Degraded-mode behaviour”If the OTLP endpoint is unreachable, the SDK’s batch exporters:
- Buffer records/spans in bounded in-memory queues and retry
- Drop data when a queue overflows; never block the daemon
- Surface export failures as
WARNlines in the daemon log (deliberately routed outside the log bridge, so a failing log export cannot feed itself)
Losing telemetry never blocks daemon work: all export is asynchronous, off the hot paths.
HUD: the live overview
Section titled “HUD: the live overview”For a real-time operator view that doesn’t require an observability backend:
shc hudThe HUD is a full-screen TUI that subscribes to the same event stream, renders node/stack/container health live, and surfaces errors as they happen. See architecture/modules/hud.
It’s the single best first-look tool when something seems wrong. On a phone it runs in the SHC Console’s TUI mode, driven by gestures: The HUD on a phone.
Common operational recipes
Section titled “Common operational recipes””A stack is unhealthy: where do I look?”
Section titled “”A stack is unhealthy: where do I look?””shc hud: visual overview, shows red where things are broken.shc logs -s <stack>: raw container logs.shc events --follow: daemon events (install, hook runs, healthcheck transitions).shc events: lifecycle events (A200100STKINS,V500320STKERR, etc.).
”A node is degraded: what’s the timeline?"
Section titled “”A node is degraded: what’s the timeline?"”shc events --node <id> --since 1hshc events --node <id> # live events from that nodeshc events --audit --since <YYYY-MM-DD> # audited actions in the window"I want metrics for just one tenant’s stacks”
Section titled “"I want metrics for just one tenant’s stacks””All container metrics carry tenant, stack, environment labels.
Query on those:
rate(container_cpu_usage_seconds_total{tenant="acme"}[5m])Related
Section titled “Related”- Disaster recovery: what a degraded control plane means and how to fix it
- Architecture: events flow: how lifecycle events propagate to subscribers
- Architecture: audit module: audit trail details
- Architecture: metrics module: metric collection internals
- OpenObserve app: turnkey OTLP target (the managed default)
- Error reference: what error codes mean