Metrics reference
Generated file, do not edit by hand. Regenerate from the repo root with
CGO_ENABLED=0 go run scripts/gen-metrics-md.go.
Every series the daemon pushes to the OpenTelemetry collector, and every queue that reports pressure. Nothing here is scraped: the daemon pushes, and no decision the daemon makes reads these series back.
Resource
Section titled “Resource”Every signal shcd pushes carries its node in the OpenTelemetry resource, which
OpenObserve shows as labels on each series: node is the node’s name (its id
when it has none), node_id, service_instance_id and host_id are its id,
and host_name is the host’s own name. Two nodes reporting the same queue are
two series, so count by (node) (shc_pressure_depth) has one group per
exporting node. A node that has not run shc init or shc node join carries
no node, node_id, service_instance_id or host_id.
Queue pressure
Section titled “Queue pressure”Each queue reports its depth, the age of its oldest waiting item, its in and out
rate and its drops or refusals as the shc_pressure_* series below, labelled
queue, and is graded green, amber or red against its thresholds. A level
rises the moment a threshold is crossed and falls only after the reading has
stayed under 80% of it for 30 seconds. shc status prints one line per queue
per node and shc doctor raises a finding for each amber or red queue.
A queue marked for one peer holds items only one other node can take. While that node’s lease is lost, the placement freeze leaves the queue out of what it decides from, so a lost node’s backlog never holds the moves that replace it.
Total: 79 queues.
| Queue | Holds | Capacity | Amber | Red | A drop raises | For one peer | What it measures |
|---|---|---|---|---|---|---|---|
acme.orders | certificates | unbounded | age 1h0m0s | - | amber | no | Certificates routes ask for that are not issued yet, cluster-wide; the age is how long the oldest has been wanted, out are wants settled or no longer asked for, refusals are orders that failed and wait out a backoff; only the ACME CA settles them, so the placement freeze leaves it out |
acme.unindexed | certificates | unbounded | age 5m0s | - | nothing | no | Certificates this node stored in the vault that the certificate index does not list yet, retried every tick while it leads |
api.<provider> | accounts | unbounded | - | - | nothing | no | API accounts held by rate limits: depth is held accounts, age the longest hold, drops provider refusals and local refusals |
api.<provider>.slots | calls | unbounded | age 15s | age 30s | amber | no | calls waiting for one of a provider account’s in-flight slots: the depth is how many wait now and the age how long the first has waited; in counts the calls that asked for a slot, out the ones that got one, and refusals the ones whose context ended while they waited |
app.dns_publishes | hosts | unbounded | age 15m0s | - | nothing | no | Platform-owned hostnames, the cluster host, module hosts and deployment hosts, whose DNS record publish failed and is retried with backoff until the provider takes it; the age is how long the oldest has gone unpublished, out are publishes that landed, drops are hosts no longer wanted or a pass that wrote nothing, and a node that stops leading stops counting the cluster’s own hosts. A host another claim holds waits for its takeover outside the queue, named by doctor app.dns-unpublished |
app.dns_releases | hosts | unbounded | age 1h0m0s | - | nothing | no | Superseded system hostnames whose DNS record release failed or left a private plane unswept, retried by every system host pass until it answers; the age is how long the oldest has waited, out are releases that answered and hosts wanted again, drops are records found not ours, and a node whose passes stop (it no longer leads) forgets them after ten minutes |
app.lease_reaps | deployments | unbounded | age 1h0m0s | - | amber | no | Deployments whose lease has expired and that the leader’s reap sweep has not removed; the age is how long ago the oldest expired, out are leases reaped, renewed or gone, refusals are reaps that failed and wait out a backoff, drops are deployments abandoned after the retry budget |
app.orphaned_roles | roles | unbounded | age 1h0m0s | - | nothing | no | Access roles on this node’s ledger that an uninstall could not delete and a live Keycloak can confirm, retried until it does; a role only a gone node’s Keycloak can confirm is app.orphaned_roles. |
app.orphaned_roles.<node> | roles | unbounded | age 1h0m0s | - | nothing | yes | Access roles on this node’s ledger that only one node can confirm gone, because it alone holds the system Keycloak deployment; only that node can end the wait |
app.parked | deployments | unbounded | - | - | amber | no | Deployments this node’s boot pass found parked in an in-flight status, left waiting for an operator while the pass only reports; the age is the oldest row’s status change, or its before-image’s start when one survives; in are rows found parked, out are rows whose settle or restore write succeeded and that left the parked status, a failed write leaves the row waiting, and drops are rows stamped unknown or whose before-image this build cannot read |
app.refinish | deployments | unbounded | age 30m0s | - | nothing | no | Deployments the leader is re-finishing for the shcd it runs, from the pass taking each until its rollout is over, and those whose re-finish or rollout failed until a retry gets them through; the age is how long the oldest has waited, out are deployments re-finished or found unchanged, and a node that stops leading forgets them after ten minutes |
app.replica_docs | deployments | unbounded | age 1h0m0s | - | nothing | no | Deployments whose replica documents a record or an uninstall could not prune from the vault, one row each in the cluster store, retried by the leader until the prune succeeds; the age is how long the oldest has waited |
backup.fire_retries | deployments | unbounded | - | - | nothing | no | Deployments a scheduled backup fire could not back up because a node did not answer in time or could not be reached, held in that fire for another attempt 2, 10 and 30 minutes on; the age is how long the oldest has waited, out are attempts that sealed, drops are deployments whose last attempt failed or failed on an error a retry cannot clear |
backup.paused | deployments | unbounded | depth 1 | - | nothing | no | Journal entries on this node for deployments a backup left paused when it died, waiting for the resume sweep to bring the deployment back and retire the entry; the age is how long the oldest has been paused, drops are entries whose deployment is gone or runs on another node now |
backup.prune | repositories | unbounded | age 6h0m0s | - | amber | no | Restic repositories a retention sweep forgot snapshots in and has not started pruning; out are repositories pruned, drops are prunes that failed, which wait for the next sweep’s forgets |
backup.quiesced | deployments | unbounded | depth 1 | - | nothing | no | Journal entries on this node for deployments a backup left quiesced when it died, waiting for the resume sweep to bring the deployment back and retire the entry; the age is how long the oldest has been quiesced, drops are entries whose deployment is gone or runs on another node now |
backup.stopped | deployments | unbounded | depth 1 | - | nothing | no | Journal entries on this node for deployments a backup left stopped when it died, waiting for the resume sweep to bring the deployment back and retire the entry; the age is how long the oldest has been stopped, drops are entries whose deployment is gone or runs on another node now |
capacity.pending | requests | unbounded | age 30m0s | - | nothing | no | Installs, updates and failovers pending for a node with room, cluster-wide; the age is how long the oldest has waited, out are requests that landed, drops are requests uninstalled or displaced by another request of the deployment; nothing expires, a node with room or the portal’s autoscaler settles them |
clone.moves | moves | unbounded | age 1h0m0s | - | nothing | no | Moves journalled on this node that a live node can finish: in flight, or left by a crash for the recovery at the next start; a move with exactly one gone node among the nodes its rollback still needs is clone.moves. |
clone.moves.<node> | moves | unbounded | age 1h0m0s | - | nothing | yes | Moves journalled on this node with exactly one gone node among the nodes its rollback still needs; only that node can end the wait |
cluster.store_tier_moves | moves | unbounded | age 5m0s | - | nothing | no | The store tier move in flight, one at most, kept by the leader’s tier pass while the move’s node is reachable; the age is how long it has been open, out are moves that ended, drops are moves cancelled because the plan no longer gave the node the move’s role, and a node that stops leading forgets it after five minutes |
clusterfs.revision_lag | revisions | unbounded | age 10s | age 1m0s | nothing | no | Revisions this node’s clusterfs is behind the store; in are store revisions this node saw, out are revisions it applied, and a materialiser never drops one |
containerbroker.log_follows | reads | 256 | depth 192, age 1m0s | - | amber | no | Console reads stamped for the clients following an incus brokered container’s timestamped log, one read’s new lines each; the depth is the furthest-behind follow, the age is the oldest read any follow has not written; a follow the capacity behind is ended (X503997CBRLAG), drops are the reads refused at that bound, and the reads of a client that left unread are retired |
cron.fires | crons | unbounded | depth 1 | - | nothing | no | Crons on this node whose last fire is older than twice their interval, still running past their timeout, or whose last two fires both ended other than ok |
daemon.memory | crossings | unbounded | age 30s | age 2m0s | nothing | no | Memory stalls of this node’s shcd unit: in counts the times its cgroup crossed memory.high (memory.events high), the line over which the kernel throttles every task in it, and the age is how long the unit has been held back, every reading in a row in which it crossed that line or all its tasks stalled on memory (memory.pressure full) for 10% of the time or more. Amber after 30s, red after 2m0s. A daemon that runs in no systemd service reports nothing |
db.local | statements | unbounded | age 5s | - | nothing | no | statements waiting for a connection of the reads this node’s own replica answers: the depth is how many wait now and the age how long the first has waited; in counts the statements that asked for one, out the ones that got one, and refusals the ones whose context ended while they waited |
db.reader | statements | unbounded | age 5s | - | nothing | no | statements waiting for a read connection: the depth is how many wait now and the age how long the first has waited; in counts the statements that asked for one, out the ones that got one, and refusals the ones whose context ended while they waited |
db.replica_missing | revisions | unbounded | depth 1 | - | nothing | no | Store revisions this node’s own replica is missing although its raft log is applied as far as the leader’s: a diverged copy, not one that is behind. While any is missing the leader answers this node’s reads. In, out and drops are not counted |
db.writer | statements | unbounded | age 5s | - | nothing | no | statements waiting for the single write connection: the depth is how many wait now and the age how long the first has waited; in counts the statements that asked for one, out the ones that got one, and refusals the ones whose context ended while they waited |
dns.commits | endpoints | unbounded | age 15m0s | - | nothing | no | DNS provider endpoints holding record writes they have not made live, because the commit after them was refused (a pfSense apply dispatcher full of queued applies) or failed; each is retried with backoff, never sooner than the wait the endpoint stated, until a commit lands; the age is how long the oldest has waited, out are commits that landed, drops are commits given up after the last attempt |
dns.providers | endpoints | unbounded | - | - | nothing | no | DNS provider API endpoints whose circuit is open: depth is the open circuits, age the longest open, drops the calls refused |
events.broadcast.<peer> | events | 256 | depth 128, age 30s | - | amber | yes | Events this node still owes one peer, queued or being retried; drops are overflow sheds, exhausted retries and everything for a peer whose lease is lost or was never held; the placement freeze leaves it out while that peer’s lease is lost |
events.bus | events | 10000 | depth 5000, age 5s | depth 9000, age 30s | red | no | Events waiting for the in-process consumer |
events.detached_writes | events | 4096 | depth 1, age 1m0s | - | amber | no | Events of coded log lines whose row the store did not take in time, kept to be written in order once it answers; each is in the journal and on the local bus already; drops are lines a full queue could not keep and rows the store refused for good |
events.pending_broadcast | events | unbounded | depth 1000, age 2m0s | depth 10000, age 15m0s | amber | no | Events in the replicated log not yet acknowledged by every peer; in, out and drops count this node’s writes, settles and prunes; the pending rows of a lost node, which the leader clears, are counted as retired and never graded |
events.subscribers | events | 256 | depth 192, age 1m0s | - | amber | no | Events buffered for this node’s event subscribers (SSE clients and in-process watchers); the depth is the fullest subscriber’s buffer, the age is the oldest event any subscriber has not taken, drops are events a full buffer lost for its subscriber or a subscriber left unread |
health.retries | deployments | unbounded | age 30m0s | - | amber | no | Deployments the health reconcile on this node failed to bring back and retries with a backoff; the age is how long the oldest has been failing, out are deployments that recovered or were reset, drops are deployments given up on after the last attempt |
incus.api | requests | unbounded | age 10m0s | age 30m0s | nothing | no | Requests this node’s shcd has sent its Incus and not had answered, including exec attaches from the exec request until every stream attaches: the age is how long Incus has answered slowly, every minute in a row whose 90th percentile answer time passed 1s (10s for a listing with every instance’s state) or that ended with a request waiting longer than that, in and out are requests sent and answered. Amber after 10m0s of slow minutes, red after 30m0s; the incus.api-latency doctor check names the answer times. A node that runs no Incus reports nothing |
incus.heavy | requests | unbounded | age 5m0s | age 15m0s | nothing | no | Heavy requests this node’s shcd waits to send its Incus: creates and deletes of instances, networks, images and custom volumes, and listings with recursion, at most 4 in flight at once (SHC-3396). The age is how long the oldest has waited; amber after 5m0s, red after 15m0s. A queue that stays up means Incus answers slowly; incus.api says how slowly |
labelcheck.unknown | labels | unbounded | - | - | nothing | no | Live labels whose probe could not answer on its latest pass, each keeping the value it last reported; the age is how long the longest has gone unanswered, in are labels that stopped answering, out are labels that answered again or left the node |
leader.loops | loops | unbounded | - | depth 1 | nothing | no | Watched control-plane loops on this node that have completed no sweep past their budget; in are loops that stalled, out are loops that swept again; on the leader a stall past the grace steps its rqlite leadership down |
log.notify | notifications | unbounded | depth 256, age 30s | - | nothing | no | Coded log lines handed to the notifier and not yet returned from it; each runs on its own goroutine, so none is refused and nothing bounds the depth |
mesh.pmtu.<device> | rounds | unbounded | age 3m0s | - | amber | no | Path MTU probe rounds on this device that are due and have not run, and how long the oldest has been due; a host nobody asked about, forgotten after the window, is retired and not a drop |
mesh.relay | pairs | unbounded | age 5m0s | - | nothing | no | Relay decisions waiting for the session worker |
node.loops | loops | unbounded | depth 1 | age 5m0s | nothing | no | Per-node reconcile loops that have completed no sweep past their budget; age is time since the last completed sweep or start; in are loops that stalled, out are loops that swept again or stopped |
nodelease.renewal | renewals | unbounded | depth 1, age 7s | age 30s | nothing | no | This node’s lease renewals the leader has not acknowledged. With no leader answering, only the misses count; once a leader answers and refuses, the age runs as well |
notification.debounce | events | unbounded | age 10m0s | - | nothing | no | Events a rule’s debounce window holds back to send as one notification; the window opens at the first event held and closes one window later, so the age is how long the oldest has been held; drops are events a notification config change cleared |
notification.send | notifications | 256 | depth 128, age 30s | depth 230, age 2m0s | amber | no | Notifications waiting for a send worker |
operation.streams | frames | 64 | depth 48, age 1m0s | - | amber | no | Progress frames buffered for the subscribers of operation streams; the depth is the fullest subscriber’s buffer, drops are frames a full buffer lost for its subscriber or a subscriber left unread |
operations.events | events | 64 | depth 48, age 1m0s | - | amber | no | Step events buffered for the client following each operation, counted while a client follows it; the depth is the fullest such buffer, drops are events a full buffer lost or an operation was forgotten with unread |
otel.relay | exports | 256 | depth 128, age 30s | depth 230, age 2m0s | amber | no | Telemetry exports relayed to this node for the managed openobserve, waiting to be forwarded; drops are the oldest evicted from a full queue (by count or by 64 MiB of bodies), exports openobserve refused for good and exports still failing five minutes after they were queued; exports queued while no node holds openobserve are counted as retired and never graded; only openobserve drains it, so the placement freeze leaves it out |
pods.early_probes | probes | unbounded | age 2m0s | age 4m0s | nothing | no | Starting containers on this node whose own health test SHC runs until it first passes, so what waits on them healthy need not wait for the runtime’s first probe; the age is how long the oldest has been probed, out are probes that passed, drops are probes that ended without a pass |
pods.pulls | pulls | unbounded | age 20m0s | age 1h0m0s | nothing | no | Image pulls of replica projects on this node asked and not ended, waiting for one of the node’s pull slots or running in one; the age is how long the oldest has been asked, out are pulls ended, drops are pulls a pass gave up before they ran |
pods.restarts | projects | unbounded | depth 1 | - | nothing | no | Replica projects on this node a restart stopped and has not brought up again; the age is how long the oldest has been down, out are projects brought up, drops are restarts a stop, a takedown or a move of the replica ended |
pods.starts | starts | unbounded | age 15m0s | age 45m0s | nothing | no | Replica starts on this node asked and not ended, waiting for one of the node’s start slots or running in one; the age is how long the oldest has been asked, out are starts ended |
resolver.dns | queries | 4096 | depth 2048, age 250ms | depth 3686, age 1s | amber | no | DNS queries read and not yet answered |
resolver.forward_waiters | queries | 131072 | depth 65536, age 2s | depth 117964, age 4s | nothing | no | DNS queries waiting on an upstream exchange an identical query started; in are queries that joined one, out are queries its answer or failure reached |
resolver.revision | revisions | unbounded | depth 1 | age 30s | nothing | no | clusterfs revisions this node’s DNS should answer from and does not; its last good answers stand; in are the times it fell behind, out the times it caught up, and it drops none |
resolver.tcp_replies | answers | 64 | depth 48, age 5s | - | amber | no | DNS answers waiting to be written to TCP clients; the depth is the fullest connection’s buffer, drops are answers a full buffer lost (its connection is cut off) or a closed connection left unwritten |
rpc.in_flight | calls | 64 | depth 48 | depth 64 | amber | no | Internal RPC calls the busiest peer has running on this node, against the per-peer cap; the age is the longest-running call of any peer, in and out are calls admitted and finished, refusals are calls turned away busy |
schedule.fires | schedules | unbounded | depth 40, age 5m0s | - | amber | no | Enabled schedules whose run time has passed and that have not fired, cluster-wide; the leader fires a batch per sweep, the age is how late the oldest is, out are schedules that fired or stopped being overdue (ran by another path, disabled, removed), refusals are fires that failed and wait out a backoff |
scheduler.replica_refinishes | deployments | unbounded | - | - | nothing | no | Deployments the leader’s engine holds to finish again because a replica of theirs prints its node’s location and that node’s location changed; the age is how long the oldest has waited, out are deployments whose documents print their nodes’ locations again |
scheduler.waiting | replicas | unbounded | age 5m0s | - | nothing | no | Replicas waiting on the scheduler engine that a live node can release: unbound, bound where their bytes no longer are, or moving, read from the store at most every 30 s; a wait only a gone node can end is scheduler.waiting. |
scheduler.waiting.<node> | replicas | unbounded | age 5m0s | - | nothing | yes | Replicas waiting on the scheduler engine that only one node can release, because their bytes are on it or a move is taking them to it; only that node can end the wait |
selfclean.room | disks | unbounded | age 10m0s | age 1h0m0s | nothing | no | disks on this node past the self-clean sweep mark that its sweeps have not brought under it |
single_writer.restarts | writers | unbounded | depth 1 | - | nothing | no | Single writers on this node a lease-loss stop took down that the node agent has not started again; the age is how long the oldest has been down, drops are writers whose container is gone or whose deployment is no longer this node’s |
storage.volume_handbacks | volumes | unbounded | age 5m0s | - | amber | no | Volumes the automatic volume failover moved off a node that came back before its deployment moved, waiting to go back to that node; the age is how long the oldest has waited, out are volumes handed back or no longer listed, refusals are hand-backs that failed and wait out a backoff of 30 s doubling to 10 min, and a node that stops leading forgets them after two minutes |
store.replica_lag | entries | unbounded | age 30s | age 2m0s | nothing | no | Raft log entries the leader has applied that this node’s copy of the store has not, on a voter or a replica: the age is how long the copy has been behind, in are entries the leader applied and out entries this node applied, as each replica check compares them. A replica that has just joined says it is catching up. A store client runs no copy and reports nothing |
task.affinity.<node> | tasks | unbounded | age 2m0s | age 10m0s | nothing | yes | Tasks pending for one node whose lease is lost or which holds no lease, and the oldest of them that is due; only that node can claim them |
task.queue | tasks | 1000 | depth 500, age 2m0s | depth 900, age 10m0s | red | no | Tasks pending or running cluster-wide, and the oldest pending task that is due |
transfer.fetch | transfers | unbounded | age 10s | age 1m0s | amber | no | Transfers waiting for a slot to pull from a peer; in are transfers that took a slot or queued for one, out are slots granted, refusals are transfers turned away busy |
transfer.serve | transfers | unbounded | age 10s | age 1m0s | amber | no | Transfers waiting for a slot to serve a peer’s pull; in are transfers that took a slot or queued for one, out are slots granted, refusals are transfers turned away busy |
transfer.stage | transfers | unbounded | age 10s | age 1m0s | amber | no | Transfers waiting for a slot to stage; in are transfers that took a slot or queued for one, out are slots granted, refusals are transfers turned away busy |
transfer.staged_bytes | bytes | unbounded | - | - | amber | no | Bytes staged on this node for peers to pull, against its staging quota; the age is the oldest handle still staged, in and out are bytes that entered and left the quota, refusals are stagings and receipts the quota turned away |
vault.providers | endpoints | unbounded | - | - | nothing | no | External vault endpoints whose circuit is open: depth is the open circuits, age the longest open, drops the calls refused |
vault.read_audit | windows | unbounded | age 1h0m0s | - | amber | no | Vault read windows open on this node: reads counted per subject and not yet settled by the flush; the age is the oldest open window, in and out are windows opened and settled, refusals are reads the full table sent to a row of their own |
volume.binds | volumes | unbounded | age 15m0s | age 1h0m0s | nothing | no | Durable volume binds of the renders on this node asked and not ended, waiting for one of the node’s render bind slots or running in one; the age is how long the oldest has been asked, out are binds ended, drops are binds a render gave up before they ran |
volume.restart_ledger | restarts | unbounded | age 10m0s | - | nothing | no | Restart records on this node’s disk: restarts of a deployment under its volume claims that have not finished and released them |
volume.restarts | deployments | unbounded | depth 1 | - | nothing | no | Deployments on this node whose writers a lease stop took down and that are not running again; the age is how long the oldest has been down, drops are deployments the leader re-decided, which stay down here |
Series
Section titled “Series”Total: 220 series.
| Name | Type | Labels | Scope | Help |
|---|---|---|---|---|
container_cpu_cfs_periods_total | counter | name, image | every node | Number of elapsed enforcement period intervals |
container_cpu_cfs_throttled_periods_total | counter | name, image | every node | Number of throttled period intervals |
container_cpu_cfs_throttled_seconds_total | counter | name, image | every node | Total time duration the container has been throttled |
container_cpu_system_seconds_total | counter | name, image | every node | Cumulative system cpu time consumed in seconds |
container_cpu_usage_seconds_total | counter | name, image | every node | Cumulative cpu time consumed in seconds |
container_cpu_user_seconds_total | counter | name, image | every node | Cumulative user cpu time consumed in seconds |
container_fs_reads_bytes_total | counter | name, device | every node | Cumulative count of bytes read |
container_fs_writes_bytes_total | counter | name, device | every node | Cumulative count of bytes written |
container_last_seen | gauge | name, image | every node | Last time a container was seen by the exporter |
container_memory_cache | gauge | name, image | every node | Number of bytes of page cache memory |
container_memory_rss | gauge | name, image | every node | Size of RSS in bytes |
container_memory_swap | gauge | name, image | every node | Container swap usage in bytes |
container_memory_usage_bytes | gauge | name, image | every node | Current memory usage in bytes including all memory regardless of when it was accessed |
container_memory_working_set_bytes | gauge | name, image | every node | Current working set in bytes |
container_network_receive_bytes_total | counter | name, interface | every node | Cumulative count of bytes received |
container_network_receive_errors_total | counter | name, interface | every node | Cumulative count of errors encountered while receiving |
container_network_receive_packets_total | counter | name, interface | every node | Cumulative count of packets received |
container_network_transmit_bytes_total | counter | name, interface | every node | Cumulative count of bytes transmitted |
container_network_transmit_errors_total | counter | name, interface | every node | Cumulative count of errors encountered while transmitting |
container_network_transmit_packets_total | counter | name, interface | every node | Cumulative count of packets transmitted |
container_spec_cpu_period | gauge | name, image | every node | CPU period of the container |
container_spec_cpu_quota | gauge | name, image | every node | CPU quota of the container |
container_spec_cpu_shares | gauge | name, image | every node | CPU share of the container |
container_spec_memory_limit_bytes | gauge | name, image | every node | Memory limit for the container |
container_start_time_seconds | gauge | name, image | every node | Start time of the container since unix epoch in seconds |
http_request_duration_seconds | histogram | method, endpoint, tenant, stack, environment | every node | HTTP request duration |
http_requests_total | counter | method, endpoint, status, tenant, stack, environment | every node | Total HTTP requests |
machine_cpu_cores | gauge | - | every node | Number of CPU cores on the machine |
machine_memory_bytes | gauge | - | every node | Amount of memory installed on the machine |
node_boot_time_seconds | gauge | - | every node | Node boot time, in unixtime |
node_cpu_seconds_total | counter | cpu, mode | every node | Seconds the CPUs spent in each mode |
node_disk_io_time_seconds_total | counter | device | every node | Total seconds spent doing I/Os |
node_disk_read_bytes_total | counter | device | every node | Total number of bytes read completed |
node_disk_reads_completed_total | counter | device | every node | Total number of reads completed successfully |
node_disk_writes_completed_total | counter | device | every node | Total number of writes completed successfully |
node_disk_written_bytes_total | counter | device | every node | Total number of bytes written completed |
node_filesystem_avail_bytes | gauge | device, fstype, mountpoint | every node | Filesystem space available to non-root users in bytes |
node_filesystem_free_bytes | gauge | device, fstype, mountpoint | every node | Filesystem free space in bytes |
node_filesystem_size_bytes | gauge | device, fstype, mountpoint | every node | Filesystem size in bytes |
node_load1 | gauge | - | every node | 1m load average |
node_load15 | gauge | - | every node | 15m load average |
node_load5 | gauge | - | every node | 5m load average |
node_memory_Buffers_bytes | gauge | - | every node | Memory information field Buffers_bytes |
node_memory_Cached_bytes | gauge | - | every node | Memory information field Cached_bytes |
node_memory_MemAvailable_bytes | gauge | - | every node | Memory information field MemAvailable_bytes |
node_memory_MemFree_bytes | gauge | - | every node | Memory information field MemFree_bytes |
node_memory_MemTotal_bytes | gauge | - | every node | Memory information field MemTotal_bytes |
node_memory_SReclaimable_bytes | gauge | - | every node | Memory information field SReclaimable_bytes |
node_memory_Slab_bytes | gauge | - | every node | Memory information field Slab_bytes |
node_memory_SwapFree_bytes | gauge | - | every node | Memory information field SwapFree_bytes |
node_memory_SwapTotal_bytes | gauge | - | every node | Memory information field SwapTotal_bytes |
node_network_receive_bytes_total | counter | device | every node | Network device statistic receive_bytes |
node_network_receive_errs_total | counter | device | every node | Network device statistic receive_errs |
node_network_receive_packets_total | counter | device | every node | Network device statistic receive_packets |
node_network_transmit_bytes_total | counter | device | every node | Network device statistic transmit_bytes |
node_network_transmit_errs_total | counter | device | every node | Network device statistic transmit_errs |
node_network_transmit_packets_total | counter | device | every node | Network device statistic transmit_packets |
node_time_seconds | gauge | - | every node | System time in seconds since epoch (1970) |
node_uname_info | gauge | machine, nodename, release, sysname, version | every node | Labeled system information as provided by the uname system call |
shc_backup_duration_seconds | histogram | tenant, stack | every node | Time taken to create backups |
shc_backups_created_total | counter | tenant, stack, status | every node | Total number of backups created |
shc_backups_restored_total | counter | tenant, stack, status | every node | Total number of backups restored |
shc_clusterfs_snapshot_bytes | gauge | - | every node | Store value bytes copied by this node’s last clusterfs build |
shc_clusterfs_snapshot_rows | gauge | - | every node | Store rows copied by this node’s last clusterfs build |
shc_clusterfs_snapshot_seconds | gauge | - | every node | Time taken to copy the store for this node’s last clusterfs build |
shc_container_block_read_bytes_total | counter | tenant, environment, stack, app, node_id, container_id, container_name, service | every node | Total bytes read from block devices (cumulative) |
shc_container_block_write_bytes_total | counter | tenant, environment, stack, app, node_id, container_id, container_name, service | every node | Total bytes written to block devices (cumulative) |
shc_container_cpu_limit_millicores | gauge | tenant, environment, stack, app, node_id, container_id, container_name, service | every node | Container CPU limit in millicores (from compose deploy.resources.limits) |
shc_container_cpu_usage_cores | gauge | tenant, environment, stack, app, node_id, container_id, container_name, service | every node | Container CPU usage in cores (fractional) |
shc_container_health_status | gauge | tenant, environment, stack, app, node_id, container_id, container_name, service | every node | Container health status (1=healthy, 0=unhealthy, -1=none) |
shc_container_memory_limit_bytes | gauge | tenant, environment, stack, app, node_id, container_id, container_name, service | every node | Container memory limit in bytes |
shc_container_memory_usage_bytes | gauge | tenant, environment, stack, app, node_id, container_id, container_name, service | every node | Container memory usage in bytes |
shc_container_memory_utilization_ratio | gauge | tenant, environment, stack, app, node_id, container_id, container_name, service | every node | Container memory utilization as ratio of limit (0-1+) |
shc_container_network_receive_bytes_total | counter | tenant, environment, stack, app, node_id, container_id, container_name, service | every node | Total bytes received by container (cumulative) |
shc_container_network_transmit_bytes_total | counter | tenant, environment, stack, app, node_id, container_id, container_name, service | every node | Total bytes transmitted by container (cumulative) |
shc_container_restart_count | gauge | tenant, environment, stack, app, node_id, container_id, container_name, service | every node | Number of container restarts |
shc_container_running | gauge | tenant, environment, stack, app, node_id, container_id, container_name, service | every node | Container running state (1=running, 0=stopped) |
shc_container_status | gauge | deployment, service, container, status | every node | Container SHC status (1=current status, check ‘status’ label) |
shc_crons_active | gauge | - | leader | Number of active (non-paused) cron jobs |
shc_crons_paused | gauge | - | leader | Number of paused cron jobs |
shc_deployment_status | gauge | deployment, status | every node | Deployment SHC status (1=current status, check ‘status’ label) |
shc_environment_cpu_scheduled_millicores | gauge | tenant, environment | every node | Environment CPU scheduled in millicores (sum of stack_name declared limits) |
shc_environment_cpu_usage_millicores | gauge | tenant, environment | every node | Environment CPU usage in millicores (sum of stacks) |
shc_environment_disk_usage_bytes | gauge | tenant, environment | every node | Environment disk usage in bytes (sum of stacks) |
shc_environment_memory_limit_bytes | gauge | tenant, environment | every node | Environment memory limit in bytes (sum of stack_name cgroup limits) |
shc_environment_memory_scheduled_bytes | gauge | tenant, environment | every node | Environment memory scheduled in bytes (sum of stack_name declared limits) |
shc_environment_memory_usage_bytes | gauge | tenant, environment | every node | Environment memory usage in bytes (sum of stacks) |
shc_event_bus_events_dropped | counter | - | every node | Events dropped due to queue full |
shc_event_bus_events_total | counter | code_prefix | every node | Total events processed by the event bus |
shc_event_bus_handler_errors | counter | handler | every node | Handler execution errors |
shc_event_bus_queue_depth | gauge | - | every node | Current number of events in the queue |
shc_event_bus_running | gauge | - | every node | Whether the event bus is running (1=yes, 0=no) |
shc_node_containers_running | gauge | node_id, address, role | every node | Number of running containers on node |
shc_node_containers_total | gauge | node_id, address, role | every node | Total number of containers on node |
shc_node_cpu_capacity_millicores | gauge | node_id, address, role, runtime | every node | Total CPU capacity on node in millicores (hardware) |
shc_node_cpu_cores | gauge | node_id, address, role | every node | Number of CPU cores on node |
shc_node_cpu_load_15m | gauge | node_id, address, role | every node | Node CPU load average (15 minutes) |
shc_node_cpu_load_1m | gauge | node_id, address, role | every node | Node CPU load average (1 minute) |
shc_node_cpu_load_5m | gauge | node_id, address, role | every node | Node CPU load average (5 minutes) |
shc_node_cpu_schedulable_millicores | gauge | node_id, address, role, runtime | every node | CPU available for scheduling in millicores (capacity - scheduled) |
shc_node_cpu_schedule_utilization_ratio | gauge | node_id, address, role, runtime | every node | Node CPU scheduling pressure (scheduled / capacity, 0-1+) |
shc_node_cpu_scheduled_millicores | gauge | node_id, address, role, runtime | every node | CPU scheduled on node in millicores (sum of compose deploy.resources.limits) |
shc_node_cpu_utilization_ratio | gauge | node_id, address, role | every node | Node CPU utilization as ratio (0-1) |
shc_node_deployment_count | gauge | node_id, address, role, runtime | every node | Number of deployments scheduled on this node |
shc_node_disk_available_bytes | gauge | node_id, address, role, mountpoint, device, fstype | every node | Available disk space in bytes |
shc_node_disk_capacity_bytes | gauge | node_id, address, role, runtime | every node | Total disk capacity on node in bytes (hardware) |
shc_node_disk_total_bytes | gauge | node_id, address, role, mountpoint, device, fstype | every node | Total disk space in bytes |
shc_node_disk_used_bytes | gauge | node_id, address, role, mountpoint, device, fstype | every node | Used disk space in bytes |
shc_node_disk_utilization_ratio | gauge | node_id, address, role, mountpoint, device, fstype | every node | Disk utilization as ratio (0-1) |
shc_node_docker_up | gauge | node_id, address, role | every node | Docker daemon status on node (1=up, 0=down) |
shc_node_memory_available_bytes | gauge | node_id, address, role | every node | Available memory on node in bytes |
shc_node_memory_capacity_bytes | gauge | node_id, address, role, runtime | every node | Total memory capacity on node in bytes (hardware) |
shc_node_memory_schedulable_bytes | gauge | node_id, address, role, runtime | every node | Memory available for scheduling in bytes (capacity - scheduled) |
shc_node_memory_schedule_utilization_ratio | gauge | node_id, address, role, runtime | every node | Node memory scheduling pressure (scheduled / capacity, 0-1+) |
shc_node_memory_scheduled_bytes | gauge | node_id, address, role, runtime | every node | Memory scheduled on node in bytes (sum of compose deploy.resources.limits) |
shc_node_memory_total_bytes | gauge | node_id, address, role | every node | Total memory on node in bytes |
shc_node_memory_used_bytes | gauge | node_id, address, role | every node | Used memory on node in bytes |
shc_node_memory_utilization_ratio | gauge | node_id, address, role | every node | Node memory utilization as ratio (0-1) |
shc_node_network_receive_bytes_total | counter | node_id, address, role, interface | every node | Total bytes received on interface (cumulative) |
shc_node_network_receive_errors_total | counter | node_id, address, role, interface | every node | Total receive errors on interface (cumulative) |
shc_node_network_receive_packets_total | counter | node_id, address, role, interface | every node | Total packets received on interface (cumulative) |
shc_node_network_transmit_bytes_total | counter | node_id, address, role, interface | every node | Total bytes transmitted on interface (cumulative) |
shc_node_network_transmit_errors_total | counter | node_id, address, role, interface | every node | Total transmit errors on interface (cumulative) |
shc_node_network_transmit_packets_total | counter | node_id, address, role, interface | every node | Total packets transmitted on interface (cumulative) |
shc_node_up | gauge | node_id, address, role | every node | Node availability status (1=up, 0=down) |
shc_objchange_audit_failures_total | counter | - | every node | Object changes written whose audit entry could not be written |
shc_pressure_depth | gauge | queue | every node | Items waiting in the queue |
shc_pressure_dropped_total | counter | queue | every node | Items dropped or refused |
shc_pressure_in_per_second | gauge | queue | every node | Items admitted per second, smoothed |
shc_pressure_in_total | counter | queue | every node | Items admitted |
shc_pressure_level | gauge | queue | every node | Pressure level (0=green, 1=amber, 2=red) |
shc_pressure_oldest_age_seconds | gauge | queue | every node | Age of the oldest waiting item |
shc_pressure_out_per_second | gauge | queue | every node | Items worked per second, smoothed |
shc_pressure_out_total | counter | queue | every node | Items worked |
shc_pressure_retired_total | counter | queue, reason | every node | Items the owner took out for the reason given, neither worked nor dropped; only a queue that retires reports it |
shc_queue_depth | gauge | - | every node | Current queue depth (PENDING + RUNNING tasks) |
shc_queue_is_leader | gauge | - | every node | Whether this node is the queue leader (1=yes, 0=no) |
shc_queue_registered_crons | gauge | - | every node | Number of cron jobs registered in the queue |
shc_queue_registered_handlers | gauge | - | every node | Number of event handlers registered in the queue |
shc_queue_rejected_total | counter | - | every node | Total tasks rejected due to backpressure (queue full) |
shc_queue_retries_total | counter | - | every node | Total task retries scheduled after failure or timeout |
shc_queue_running | gauge | - | every node | Whether the queue is running (1=yes, 0=no) |
shc_queue_running_tasks | gauge | - | every node | Number of tasks currently executing in this node’s worker pool |
shc_queue_timeouts_total | counter | - | every node | Total tasks terminated due to max_runtime_seconds |
shc_quota_apps_limit | gauge | tenant, environment, stack | every node | Maximum number of apps allowed |
shc_quota_apps_used | gauge | tenant, environment, stack | every node | Number of apps in use |
shc_quota_cpu_limit | gauge | tenant, environment, stack | every node | CPU quota limit in millicores |
shc_quota_cpu_used | gauge | tenant, environment, stack | every node | CPU usage in millicores (allocated) |
shc_quota_cpu_utilization_ratio | gauge | tenant, environment, stack | every node | CPU utilization as ratio of limit (0-1+) |
shc_quota_disk_limit_bytes | gauge | tenant, environment, stack | every node | Disk quota limit in bytes |
shc_quota_disk_used_bytes | gauge | tenant, environment, stack | every node | Disk usage in bytes (measured) |
shc_quota_disk_utilization_ratio | gauge | tenant, environment, stack | every node | Disk utilization as ratio of limit (0-1+) |
shc_quota_environments_limit | gauge | tenant, environment, stack | every node | Maximum number of environments allowed |
shc_quota_environments_used | gauge | tenant, environment, stack | every node | Number of environments in use |
shc_quota_memory_limit_bytes | gauge | tenant, environment, stack | every node | Memory quota limit in bytes |
shc_quota_memory_used_bytes | gauge | tenant, environment, stack | every node | Memory usage in bytes (allocated) |
shc_quota_memory_utilization_ratio | gauge | tenant, environment, stack | every node | Memory utilization as ratio of limit (0-1+) |
shc_quota_over_limit | gauge | tenant, environment, stack | every node | Indicates if scope is over any quota limit (1=over, 0=ok) |
shc_quota_stacks_limit | gauge | tenant, environment, stack | every node | Maximum number of stacks allowed |
shc_quota_stacks_used | gauge | tenant, environment, stack | every node | Number of stacks in use |
shc_quota_vdeployments_limit | gauge | tenant, environment, stack | every node | Maximum number of vdeployments allowed |
shc_quota_vdeployments_used | gauge | tenant, environment, stack | every node | Number of vdeployments installed |
shc_rpc_busy_refusals_total | counter | peer | every node | Internal RPC calls refused busy at the per-peer in-flight cap |
shc_rpc_in_flight | gauge | peer | every node | Internal RPC calls a peer has running on this node |
shc_service_cpu_limit_millicores | gauge | tenant, environment, stack, app, service, node_id | every node | Service CPU limit in millicores (from compose deploy.resources.limits) |
shc_service_cpu_request_millicores | gauge | tenant, environment, stack, app, service, node_id | every node | Service CPU reservation in millicores (from compose deploy.resources.reservations) |
shc_service_cpu_usage_millicores | gauge | tenant, environment, stack, app, service, node_id | every node | Service CPU usage in millicores (sum of containers) |
shc_service_memory_limit_bytes | gauge | tenant, environment, stack, app, service, node_id | every node | Service memory limit in bytes (sum of container cgroup limits) |
shc_service_memory_limit_declared_bytes | gauge | tenant, environment, stack, app, service, node_id | every node | Service memory limit in bytes (from compose deploy.resources.limits) |
shc_service_memory_request_bytes | gauge | tenant, environment, stack, app, service, node_id | every node | Service memory reservation in bytes (from compose deploy.resources.reservations) |
shc_service_memory_usage_bytes | gauge | tenant, environment, stack, app, service, node_id | every node | Service memory usage in bytes (sum of containers) |
shc_service_status | gauge | deployment, service, status | every node | Service SHC status (1=current status, check ‘status’ label) |
shc_sse_broker_channels | gauge | - | every node | Number of active SSE broker channels |
shc_sse_broker_events_dropped_total | counter | - | every node | Events dropped due to slow subscriber queues |
shc_sse_broker_events_replayed_total | counter | - | every node | Events replayed from buffer on reconnection |
shc_sse_broker_events_total | counter | event_type | every node | Total events published through SSE broker |
shc_sse_broker_session_duration_seconds | histogram | - | every node | SSE subscriber session duration in seconds |
shc_sse_broker_subscribers | gauge | - | every node | Number of active SSE broker subscribers |
shc_sse_connections | gauge | operation_type | every node | Number of active SSE connections |
shc_sse_event_latency_seconds | histogram | event_type | every node | Time from event creation to SSE emission |
shc_sse_events_buffered | gauge | operation_id | every node | Events currently buffered per operation |
shc_sse_events_dropped_total | counter | operation_id | every node | Events dropped due to buffer overflow |
shc_sse_events_emitted_total | counter | event_type | every node | Total events emitted to SSE streams |
shc_sse_memory_bytes | gauge | - | every node | Estimated memory usage for SSE buffers |
shc_sse_operations | gauge | status | every node | Number of tracked operations |
shc_sse_slow_clients_total | counter | - | every node | Number of slow client detections (gap events emitted) |
shc_stack_cpu_limit_millicores | gauge | tenant, environment, stack, app, node_id, runtime | every node | Stack CPU limit in millicores (sum of service limits) |
shc_stack_cpu_request_millicores | gauge | tenant, environment, stack, app, node_id, runtime | every node | Stack CPU reservation in millicores (sum of service reservations) |
shc_stack_cpu_usage_millicores | gauge | tenant, environment, stack, app, node_id, runtime | every node | Stack CPU usage in millicores (sum of services) |
shc_stack_disk_usage_bytes | gauge | tenant, environment, stack, app, node_id, runtime | every node | Stack disk usage in bytes (from quota disk measurement) |
shc_stack_memory_limit_bytes | gauge | tenant, environment, stack, app, node_id, runtime | every node | Stack memory limit in bytes (sum of service cgroup limits) |
shc_stack_memory_limit_declared_bytes | gauge | tenant, environment, stack, app, node_id, runtime | every node | Stack memory limit in bytes (sum of service limits) |
shc_stack_memory_request_bytes | gauge | tenant, environment, stack, app, node_id, runtime | every node | Stack memory reservation in bytes (sum of service reservations) |
shc_stack_memory_usage_bytes | gauge | tenant, environment, stack, app, node_id, runtime | every node | Stack memory usage in bytes (sum of services) |
shc_stacks_active | gauge | tenant, environment | every node | Number of currently active stacks |
shc_stacks_deployed_total | counter | tenant, app, environment | every node | Total number of stacks deployed |
shc_subsystem_state | gauge | subsystem | every node | Subsystem state (1=running, 0.5=degraded, 0=stopped, -1=failed) |
shc_tasks_total | gauge | status | leader | Number of tasks by status |
shc_tenant_cpu_scheduled_millicores | gauge | tenant | every node | Tenant CPU scheduled in millicores (sum of declared limits) |
shc_tenant_cpu_usage_millicores | gauge | tenant | every node | Tenant CPU usage in millicores (sum of environments) |
shc_tenant_disk_usage_bytes | gauge | tenant | every node | Tenant disk usage in bytes (sum of environments) |
shc_tenant_memory_limit_bytes | gauge | tenant | every node | Tenant memory limit in bytes (sum of environment cgroup limits) |
shc_tenant_memory_scheduled_bytes | gauge | tenant | every node | Tenant memory scheduled in bytes (sum of declared limits) |
shc_tenant_memory_usage_bytes | gauge | tenant | every node | Tenant memory usage in bytes (sum of environments) |
shc_transfer_active | gauge | side, class | every node | Transfers running on this node: serving (side=serve), pulling (side=fetch) or staging (side=stage) |
shc_transfer_busy_refusals_total | counter | side, class | every node | Transfers refused busy: queue full or no slot before the caller’s deadline |
shc_transfer_bytes_per_second | gauge | direction | every node | Transfer throughput over the last tick, received (direction=in) or served (direction=out) |
shc_transfer_bytes_total | counter | direction | every node | Transfer bytes received (direction=in) or served (direction=out) |
shc_transfer_hash_failures_total | counter | - | every node | Pulls whose sha256 did not match the holder’s |
shc_transfer_oldest_wait_seconds | gauge | side, class | every node | Age of the oldest transfer waiting for a slot |
shc_transfer_peer_active | gauge | side, class, peer | every node | Transfers running on this node per peer |
shc_transfer_quota_bytes | gauge | - | every node | This node’s staging quota |
shc_transfer_quota_refusals_total | counter | - | every node | Staging or receiving refused by the quota or the disk’s free space |
shc_transfer_staged_bytes | gauge | - | every node | Bytes staged on this node for peers to pull |
shc_transfer_sweeps_total | counter | - | every node | Expired staged handles and partial pulls swept |
shc_transfer_waiters | gauge | side, class | every node | Transfers waiting for a slot |
shc_vault_operations_total | counter | tenant, operation, status | every node | Total vault operations |
shc_vault_secrets_total | gauge | tenant, scope | every node | Total number of secrets in vault |
shc_workers_leader | gauge | - | leader | Number of workers with leader status |
shc_workers_total | gauge | - | leader | Number of registered workers |