Skip to content
SHC Docs

Metrics reference

Generated file, do not edit by hand. Regenerate from the repo root with CGO_ENABLED=0 go run scripts/gen-metrics-md.go.

Every series the daemon pushes to the OpenTelemetry collector, and every queue that reports pressure. Nothing here is scraped: the daemon pushes, and no decision the daemon makes reads these series back.

Every signal shcd pushes carries its node in the OpenTelemetry resource, which OpenObserve shows as labels on each series: node is the node’s name (its id when it has none), node_id, service_instance_id and host_id are its id, and host_name is the host’s own name. Two nodes reporting the same queue are two series, so count by (node) (shc_pressure_depth) has one group per exporting node. A node that has not run shc init or shc node join carries no node, node_id, service_instance_id or host_id.

Each queue reports its depth, the age of its oldest waiting item, its in and out rate and its drops or refusals as the shc_pressure_* series below, labelled queue, and is graded green, amber or red against its thresholds. A level rises the moment a threshold is crossed and falls only after the reading has stayed under 80% of it for 30 seconds. shc status prints one line per queue per node and shc doctor raises a finding for each amber or red queue.

A queue marked for one peer holds items only one other node can take. While that node’s lease is lost, the placement freeze leaves the queue out of what it decides from, so a lost node’s backlog never holds the moves that replace it.

Total: 79 queues.

QueueHoldsCapacityAmberRedA drop raisesFor one peerWhat it measures
acme.orderscertificatesunboundedage 1h0m0s-ambernoCertificates routes ask for that are not issued yet, cluster-wide; the age is how long the oldest has been wanted, out are wants settled or no longer asked for, refusals are orders that failed and wait out a backoff; only the ACME CA settles them, so the placement freeze leaves it out
acme.unindexedcertificatesunboundedage 5m0s-nothingnoCertificates this node stored in the vault that the certificate index does not list yet, retried every tick while it leads
api.<provider>accountsunbounded--nothingnoAPI accounts held by rate limits: depth is held accounts, age the longest hold, drops provider refusals and local refusals
api.<provider>.slotscallsunboundedage 15sage 30sambernocalls waiting for one of a provider account’s in-flight slots: the depth is how many wait now and the age how long the first has waited; in counts the calls that asked for a slot, out the ones that got one, and refusals the ones whose context ended while they waited
app.dns_publisheshostsunboundedage 15m0s-nothingnoPlatform-owned hostnames, the cluster host, module hosts and deployment hosts, whose DNS record publish failed and is retried with backoff until the provider takes it; the age is how long the oldest has gone unpublished, out are publishes that landed, drops are hosts no longer wanted or a pass that wrote nothing, and a node that stops leading stops counting the cluster’s own hosts. A host another claim holds waits for its takeover outside the queue, named by doctor app.dns-unpublished
app.dns_releaseshostsunboundedage 1h0m0s-nothingnoSuperseded system hostnames whose DNS record release failed or left a private plane unswept, retried by every system host pass until it answers; the age is how long the oldest has waited, out are releases that answered and hosts wanted again, drops are records found not ours, and a node whose passes stop (it no longer leads) forgets them after ten minutes
app.lease_reapsdeploymentsunboundedage 1h0m0s-ambernoDeployments whose lease has expired and that the leader’s reap sweep has not removed; the age is how long ago the oldest expired, out are leases reaped, renewed or gone, refusals are reaps that failed and wait out a backoff, drops are deployments abandoned after the retry budget
app.orphaned_rolesrolesunboundedage 1h0m0s-nothingnoAccess roles on this node’s ledger that an uninstall could not delete and a live Keycloak can confirm, retried until it does; a role only a gone node’s Keycloak can confirm is app.orphaned_roles. instead, and none is counted while the cluster has no system Keycloak deployed; the age is how long the oldest has waited, out are roles confirmed gone, and the ledger never gives one up
app.orphaned_roles.<node>rolesunboundedage 1h0m0s-nothingyesAccess roles on this node’s ledger that only one node can confirm gone, because it alone holds the system Keycloak deployment; only that node can end the wait
app.parkeddeploymentsunbounded--ambernoDeployments this node’s boot pass found parked in an in-flight status, left waiting for an operator while the pass only reports; the age is the oldest row’s status change, or its before-image’s start when one survives; in are rows found parked, out are rows whose settle or restore write succeeded and that left the parked status, a failed write leaves the row waiting, and drops are rows stamped unknown or whose before-image this build cannot read
app.refinishdeploymentsunboundedage 30m0s-nothingnoDeployments the leader is re-finishing for the shcd it runs, from the pass taking each until its rollout is over, and those whose re-finish or rollout failed until a retry gets them through; the age is how long the oldest has waited, out are deployments re-finished or found unchanged, and a node that stops leading forgets them after ten minutes
app.replica_docsdeploymentsunboundedage 1h0m0s-nothingnoDeployments whose replica documents a record or an uninstall could not prune from the vault, one row each in the cluster store, retried by the leader until the prune succeeds; the age is how long the oldest has waited
backup.fire_retriesdeploymentsunbounded--nothingnoDeployments a scheduled backup fire could not back up because a node did not answer in time or could not be reached, held in that fire for another attempt 2, 10 and 30 minutes on; the age is how long the oldest has waited, out are attempts that sealed, drops are deployments whose last attempt failed or failed on an error a retry cannot clear
backup.pauseddeploymentsunboundeddepth 1-nothingnoJournal entries on this node for deployments a backup left paused when it died, waiting for the resume sweep to bring the deployment back and retire the entry; the age is how long the oldest has been paused, drops are entries whose deployment is gone or runs on another node now
backup.prunerepositoriesunboundedage 6h0m0s-ambernoRestic repositories a retention sweep forgot snapshots in and has not started pruning; out are repositories pruned, drops are prunes that failed, which wait for the next sweep’s forgets
backup.quiesceddeploymentsunboundeddepth 1-nothingnoJournal entries on this node for deployments a backup left quiesced when it died, waiting for the resume sweep to bring the deployment back and retire the entry; the age is how long the oldest has been quiesced, drops are entries whose deployment is gone or runs on another node now
backup.stoppeddeploymentsunboundeddepth 1-nothingnoJournal entries on this node for deployments a backup left stopped when it died, waiting for the resume sweep to bring the deployment back and retire the entry; the age is how long the oldest has been stopped, drops are entries whose deployment is gone or runs on another node now
capacity.pendingrequestsunboundedage 30m0s-nothingnoInstalls, updates and failovers pending for a node with room, cluster-wide; the age is how long the oldest has waited, out are requests that landed, drops are requests uninstalled or displaced by another request of the deployment; nothing expires, a node with room or the portal’s autoscaler settles them
clone.movesmovesunboundedage 1h0m0s-nothingnoMoves journalled on this node that a live node can finish: in flight, or left by a crash for the recovery at the next start; a move with exactly one gone node among the nodes its rollback still needs is clone.moves. instead; the age is how long ago the oldest began, out are moves that finished or were put back, drops are records already counted here that the recovery could not read and set aside
clone.moves.<node>movesunboundedage 1h0m0s-nothingyesMoves journalled on this node with exactly one gone node among the nodes its rollback still needs; only that node can end the wait
cluster.store_tier_movesmovesunboundedage 5m0s-nothingnoThe store tier move in flight, one at most, kept by the leader’s tier pass while the move’s node is reachable; the age is how long it has been open, out are moves that ended, drops are moves cancelled because the plan no longer gave the node the move’s role, and a node that stops leading forgets it after five minutes
clusterfs.revision_lagrevisionsunboundedage 10sage 1m0snothingnoRevisions this node’s clusterfs is behind the store; in are store revisions this node saw, out are revisions it applied, and a materialiser never drops one
containerbroker.log_followsreads256depth 192, age 1m0s-ambernoConsole reads stamped for the clients following an incus brokered container’s timestamped log, one read’s new lines each; the depth is the furthest-behind follow, the age is the oldest read any follow has not written; a follow the capacity behind is ended (X503997CBRLAG), drops are the reads refused at that bound, and the reads of a client that left unread are retired
cron.firescronsunboundeddepth 1-nothingnoCrons on this node whose last fire is older than twice their interval, still running past their timeout, or whose last two fires both ended other than ok
daemon.memorycrossingsunboundedage 30sage 2m0snothingnoMemory stalls of this node’s shcd unit: in counts the times its cgroup crossed memory.high (memory.events high), the line over which the kernel throttles every task in it, and the age is how long the unit has been held back, every reading in a row in which it crossed that line or all its tasks stalled on memory (memory.pressure full) for 10% of the time or more. Amber after 30s, red after 2m0s. A daemon that runs in no systemd service reports nothing
db.localstatementsunboundedage 5s-nothingnostatements waiting for a connection of the reads this node’s own replica answers: the depth is how many wait now and the age how long the first has waited; in counts the statements that asked for one, out the ones that got one, and refusals the ones whose context ended while they waited
db.readerstatementsunboundedage 5s-nothingnostatements waiting for a read connection: the depth is how many wait now and the age how long the first has waited; in counts the statements that asked for one, out the ones that got one, and refusals the ones whose context ended while they waited
db.replica_missingrevisionsunboundeddepth 1-nothingnoStore revisions this node’s own replica is missing although its raft log is applied as far as the leader’s: a diverged copy, not one that is behind. While any is missing the leader answers this node’s reads. In, out and drops are not counted
db.writerstatementsunboundedage 5s-nothingnostatements waiting for the single write connection: the depth is how many wait now and the age how long the first has waited; in counts the statements that asked for one, out the ones that got one, and refusals the ones whose context ended while they waited
dns.commitsendpointsunboundedage 15m0s-nothingnoDNS provider endpoints holding record writes they have not made live, because the commit after them was refused (a pfSense apply dispatcher full of queued applies) or failed; each is retried with backoff, never sooner than the wait the endpoint stated, until a commit lands; the age is how long the oldest has waited, out are commits that landed, drops are commits given up after the last attempt
dns.providersendpointsunbounded--nothingnoDNS provider API endpoints whose circuit is open: depth is the open circuits, age the longest open, drops the calls refused
events.broadcast.<peer>events256depth 128, age 30s-amberyesEvents this node still owes one peer, queued or being retried; drops are overflow sheds, exhausted retries and everything for a peer whose lease is lost or was never held; the placement freeze leaves it out while that peer’s lease is lost
events.busevents10000depth 5000, age 5sdepth 9000, age 30srednoEvents waiting for the in-process consumer
events.detached_writesevents4096depth 1, age 1m0s-ambernoEvents of coded log lines whose row the store did not take in time, kept to be written in order once it answers; each is in the journal and on the local bus already; drops are lines a full queue could not keep and rows the store refused for good
events.pending_broadcasteventsunboundeddepth 1000, age 2m0sdepth 10000, age 15m0sambernoEvents in the replicated log not yet acknowledged by every peer; in, out and drops count this node’s writes, settles and prunes; the pending rows of a lost node, which the leader clears, are counted as retired and never graded
events.subscribersevents256depth 192, age 1m0s-ambernoEvents buffered for this node’s event subscribers (SSE clients and in-process watchers); the depth is the fullest subscriber’s buffer, the age is the oldest event any subscriber has not taken, drops are events a full buffer lost for its subscriber or a subscriber left unread
health.retriesdeploymentsunboundedage 30m0s-ambernoDeployments the health reconcile on this node failed to bring back and retries with a backoff; the age is how long the oldest has been failing, out are deployments that recovered or were reset, drops are deployments given up on after the last attempt
incus.apirequestsunboundedage 10m0sage 30m0snothingnoRequests this node’s shcd has sent its Incus and not had answered, including exec attaches from the exec request until every stream attaches: the age is how long Incus has answered slowly, every minute in a row whose 90th percentile answer time passed 1s (10s for a listing with every instance’s state) or that ended with a request waiting longer than that, in and out are requests sent and answered. Amber after 10m0s of slow minutes, red after 30m0s; the incus.api-latency doctor check names the answer times. A node that runs no Incus reports nothing
incus.heavyrequestsunboundedage 5m0sage 15m0snothingnoHeavy requests this node’s shcd waits to send its Incus: creates and deletes of instances, networks, images and custom volumes, and listings with recursion, at most 4 in flight at once (SHC-3396). The age is how long the oldest has waited; amber after 5m0s, red after 15m0s. A queue that stays up means Incus answers slowly; incus.api says how slowly
labelcheck.unknownlabelsunbounded--nothingnoLive labels whose probe could not answer on its latest pass, each keeping the value it last reported; the age is how long the longest has gone unanswered, in are labels that stopped answering, out are labels that answered again or left the node
leader.loopsloopsunbounded-depth 1nothingnoWatched control-plane loops on this node that have completed no sweep past their budget; in are loops that stalled, out are loops that swept again; on the leader a stall past the grace steps its rqlite leadership down
log.notifynotificationsunboundeddepth 256, age 30s-nothingnoCoded log lines handed to the notifier and not yet returned from it; each runs on its own goroutine, so none is refused and nothing bounds the depth
mesh.pmtu.<device>roundsunboundedage 3m0s-ambernoPath MTU probe rounds on this device that are due and have not run, and how long the oldest has been due; a host nobody asked about, forgotten after the window, is retired and not a drop
mesh.relaypairsunboundedage 5m0s-nothingnoRelay decisions waiting for the session worker
node.loopsloopsunboundeddepth 1age 5m0snothingnoPer-node reconcile loops that have completed no sweep past their budget; age is time since the last completed sweep or start; in are loops that stalled, out are loops that swept again or stopped
nodelease.renewalrenewalsunboundeddepth 1, age 7sage 30snothingnoThis node’s lease renewals the leader has not acknowledged. With no leader answering, only the misses count; once a leader answers and refuses, the age runs as well
notification.debounceeventsunboundedage 10m0s-nothingnoEvents a rule’s debounce window holds back to send as one notification; the window opens at the first event held and closes one window later, so the age is how long the oldest has been held; drops are events a notification config change cleared
notification.sendnotifications256depth 128, age 30sdepth 230, age 2m0sambernoNotifications waiting for a send worker
operation.streamsframes64depth 48, age 1m0s-ambernoProgress frames buffered for the subscribers of operation streams; the depth is the fullest subscriber’s buffer, drops are frames a full buffer lost for its subscriber or a subscriber left unread
operations.eventsevents64depth 48, age 1m0s-ambernoStep events buffered for the client following each operation, counted while a client follows it; the depth is the fullest such buffer, drops are events a full buffer lost or an operation was forgotten with unread
otel.relayexports256depth 128, age 30sdepth 230, age 2m0sambernoTelemetry exports relayed to this node for the managed openobserve, waiting to be forwarded; drops are the oldest evicted from a full queue (by count or by 64 MiB of bodies), exports openobserve refused for good and exports still failing five minutes after they were queued; exports queued while no node holds openobserve are counted as retired and never graded; only openobserve drains it, so the placement freeze leaves it out
pods.early_probesprobesunboundedage 2m0sage 4m0snothingnoStarting containers on this node whose own health test SHC runs until it first passes, so what waits on them healthy need not wait for the runtime’s first probe; the age is how long the oldest has been probed, out are probes that passed, drops are probes that ended without a pass
pods.pullspullsunboundedage 20m0sage 1h0m0snothingnoImage pulls of replica projects on this node asked and not ended, waiting for one of the node’s pull slots or running in one; the age is how long the oldest has been asked, out are pulls ended, drops are pulls a pass gave up before they ran
pods.restartsprojectsunboundeddepth 1-nothingnoReplica projects on this node a restart stopped and has not brought up again; the age is how long the oldest has been down, out are projects brought up, drops are restarts a stop, a takedown or a move of the replica ended
pods.startsstartsunboundedage 15m0sage 45m0snothingnoReplica starts on this node asked and not ended, waiting for one of the node’s start slots or running in one; the age is how long the oldest has been asked, out are starts ended
resolver.dnsqueries4096depth 2048, age 250msdepth 3686, age 1sambernoDNS queries read and not yet answered
resolver.forward_waitersqueries131072depth 65536, age 2sdepth 117964, age 4snothingnoDNS queries waiting on an upstream exchange an identical query started; in are queries that joined one, out are queries its answer or failure reached
resolver.revisionrevisionsunboundeddepth 1age 30snothingnoclusterfs revisions this node’s DNS should answer from and does not; its last good answers stand; in are the times it fell behind, out the times it caught up, and it drops none
resolver.tcp_repliesanswers64depth 48, age 5s-ambernoDNS answers waiting to be written to TCP clients; the depth is the fullest connection’s buffer, drops are answers a full buffer lost (its connection is cut off) or a closed connection left unwritten
rpc.in_flightcalls64depth 48depth 64ambernoInternal RPC calls the busiest peer has running on this node, against the per-peer cap; the age is the longest-running call of any peer, in and out are calls admitted and finished, refusals are calls turned away busy
schedule.firesschedulesunboundeddepth 40, age 5m0s-ambernoEnabled schedules whose run time has passed and that have not fired, cluster-wide; the leader fires a batch per sweep, the age is how late the oldest is, out are schedules that fired or stopped being overdue (ran by another path, disabled, removed), refusals are fires that failed and wait out a backoff
scheduler.replica_refinishesdeploymentsunbounded--nothingnoDeployments the leader’s engine holds to finish again because a replica of theirs prints its node’s location and that node’s location changed; the age is how long the oldest has waited, out are deployments whose documents print their nodes’ locations again
scheduler.waitingreplicasunboundedage 5m0s-nothingnoReplicas waiting on the scheduler engine that a live node can release: unbound, bound where their bytes no longer are, or moving, read from the store at most every 30 s; a wait only a gone node can end is scheduler.waiting. instead; the age is how long the oldest has waited since this node first saw it, in and out are replicas seen to start and stop waiting between two reads
scheduler.waiting.<node>replicasunboundedage 5m0s-nothingyesReplicas waiting on the scheduler engine that only one node can release, because their bytes are on it or a move is taking them to it; only that node can end the wait
selfclean.roomdisksunboundedage 10m0sage 1h0m0snothingnodisks on this node past the self-clean sweep mark that its sweeps have not brought under it
single_writer.restartswritersunboundeddepth 1-nothingnoSingle writers on this node a lease-loss stop took down that the node agent has not started again; the age is how long the oldest has been down, drops are writers whose container is gone or whose deployment is no longer this node’s
storage.volume_handbacksvolumesunboundedage 5m0s-ambernoVolumes the automatic volume failover moved off a node that came back before its deployment moved, waiting to go back to that node; the age is how long the oldest has waited, out are volumes handed back or no longer listed, refusals are hand-backs that failed and wait out a backoff of 30 s doubling to 10 min, and a node that stops leading forgets them after two minutes
store.replica_lagentriesunboundedage 30sage 2m0snothingnoRaft log entries the leader has applied that this node’s copy of the store has not, on a voter or a replica: the age is how long the copy has been behind, in are entries the leader applied and out entries this node applied, as each replica check compares them. A replica that has just joined says it is catching up. A store client runs no copy and reports nothing
task.affinity.<node>tasksunboundedage 2m0sage 10m0snothingyesTasks pending for one node whose lease is lost or which holds no lease, and the oldest of them that is due; only that node can claim them
task.queuetasks1000depth 500, age 2m0sdepth 900, age 10m0srednoTasks pending or running cluster-wide, and the oldest pending task that is due
transfer.fetchtransfersunboundedage 10sage 1m0sambernoTransfers waiting for a slot to pull from a peer; in are transfers that took a slot or queued for one, out are slots granted, refusals are transfers turned away busy
transfer.servetransfersunboundedage 10sage 1m0sambernoTransfers waiting for a slot to serve a peer’s pull; in are transfers that took a slot or queued for one, out are slots granted, refusals are transfers turned away busy
transfer.stagetransfersunboundedage 10sage 1m0sambernoTransfers waiting for a slot to stage; in are transfers that took a slot or queued for one, out are slots granted, refusals are transfers turned away busy
transfer.staged_bytesbytesunbounded--ambernoBytes staged on this node for peers to pull, against its staging quota; the age is the oldest handle still staged, in and out are bytes that entered and left the quota, refusals are stagings and receipts the quota turned away
vault.providersendpointsunbounded--nothingnoExternal vault endpoints whose circuit is open: depth is the open circuits, age the longest open, drops the calls refused
vault.read_auditwindowsunboundedage 1h0m0s-ambernoVault read windows open on this node: reads counted per subject and not yet settled by the flush; the age is the oldest open window, in and out are windows opened and settled, refusals are reads the full table sent to a row of their own
volume.bindsvolumesunboundedage 15m0sage 1h0m0snothingnoDurable volume binds of the renders on this node asked and not ended, waiting for one of the node’s render bind slots or running in one; the age is how long the oldest has been asked, out are binds ended, drops are binds a render gave up before they ran
volume.restart_ledgerrestartsunboundedage 10m0s-nothingnoRestart records on this node’s disk: restarts of a deployment under its volume claims that have not finished and released them
volume.restartsdeploymentsunboundeddepth 1-nothingnoDeployments on this node whose writers a lease stop took down and that are not running again; the age is how long the oldest has been down, drops are deployments the leader re-decided, which stay down here

Total: 220 series.

NameTypeLabelsScopeHelp
container_cpu_cfs_periods_totalcountername, imageevery nodeNumber of elapsed enforcement period intervals
container_cpu_cfs_throttled_periods_totalcountername, imageevery nodeNumber of throttled period intervals
container_cpu_cfs_throttled_seconds_totalcountername, imageevery nodeTotal time duration the container has been throttled
container_cpu_system_seconds_totalcountername, imageevery nodeCumulative system cpu time consumed in seconds
container_cpu_usage_seconds_totalcountername, imageevery nodeCumulative cpu time consumed in seconds
container_cpu_user_seconds_totalcountername, imageevery nodeCumulative user cpu time consumed in seconds
container_fs_reads_bytes_totalcountername, deviceevery nodeCumulative count of bytes read
container_fs_writes_bytes_totalcountername, deviceevery nodeCumulative count of bytes written
container_last_seengaugename, imageevery nodeLast time a container was seen by the exporter
container_memory_cachegaugename, imageevery nodeNumber of bytes of page cache memory
container_memory_rssgaugename, imageevery nodeSize of RSS in bytes
container_memory_swapgaugename, imageevery nodeContainer swap usage in bytes
container_memory_usage_bytesgaugename, imageevery nodeCurrent memory usage in bytes including all memory regardless of when it was accessed
container_memory_working_set_bytesgaugename, imageevery nodeCurrent working set in bytes
container_network_receive_bytes_totalcountername, interfaceevery nodeCumulative count of bytes received
container_network_receive_errors_totalcountername, interfaceevery nodeCumulative count of errors encountered while receiving
container_network_receive_packets_totalcountername, interfaceevery nodeCumulative count of packets received
container_network_transmit_bytes_totalcountername, interfaceevery nodeCumulative count of bytes transmitted
container_network_transmit_errors_totalcountername, interfaceevery nodeCumulative count of errors encountered while transmitting
container_network_transmit_packets_totalcountername, interfaceevery nodeCumulative count of packets transmitted
container_spec_cpu_periodgaugename, imageevery nodeCPU period of the container
container_spec_cpu_quotagaugename, imageevery nodeCPU quota of the container
container_spec_cpu_sharesgaugename, imageevery nodeCPU share of the container
container_spec_memory_limit_bytesgaugename, imageevery nodeMemory limit for the container
container_start_time_secondsgaugename, imageevery nodeStart time of the container since unix epoch in seconds
http_request_duration_secondshistogrammethod, endpoint, tenant, stack, environmentevery nodeHTTP request duration
http_requests_totalcountermethod, endpoint, status, tenant, stack, environmentevery nodeTotal HTTP requests
machine_cpu_coresgauge-every nodeNumber of CPU cores on the machine
machine_memory_bytesgauge-every nodeAmount of memory installed on the machine
node_boot_time_secondsgauge-every nodeNode boot time, in unixtime
node_cpu_seconds_totalcountercpu, modeevery nodeSeconds the CPUs spent in each mode
node_disk_io_time_seconds_totalcounterdeviceevery nodeTotal seconds spent doing I/Os
node_disk_read_bytes_totalcounterdeviceevery nodeTotal number of bytes read completed
node_disk_reads_completed_totalcounterdeviceevery nodeTotal number of reads completed successfully
node_disk_writes_completed_totalcounterdeviceevery nodeTotal number of writes completed successfully
node_disk_written_bytes_totalcounterdeviceevery nodeTotal number of bytes written completed
node_filesystem_avail_bytesgaugedevice, fstype, mountpointevery nodeFilesystem space available to non-root users in bytes
node_filesystem_free_bytesgaugedevice, fstype, mountpointevery nodeFilesystem free space in bytes
node_filesystem_size_bytesgaugedevice, fstype, mountpointevery nodeFilesystem size in bytes
node_load1gauge-every node1m load average
node_load15gauge-every node15m load average
node_load5gauge-every node5m load average
node_memory_Buffers_bytesgauge-every nodeMemory information field Buffers_bytes
node_memory_Cached_bytesgauge-every nodeMemory information field Cached_bytes
node_memory_MemAvailable_bytesgauge-every nodeMemory information field MemAvailable_bytes
node_memory_MemFree_bytesgauge-every nodeMemory information field MemFree_bytes
node_memory_MemTotal_bytesgauge-every nodeMemory information field MemTotal_bytes
node_memory_SReclaimable_bytesgauge-every nodeMemory information field SReclaimable_bytes
node_memory_Slab_bytesgauge-every nodeMemory information field Slab_bytes
node_memory_SwapFree_bytesgauge-every nodeMemory information field SwapFree_bytes
node_memory_SwapTotal_bytesgauge-every nodeMemory information field SwapTotal_bytes
node_network_receive_bytes_totalcounterdeviceevery nodeNetwork device statistic receive_bytes
node_network_receive_errs_totalcounterdeviceevery nodeNetwork device statistic receive_errs
node_network_receive_packets_totalcounterdeviceevery nodeNetwork device statistic receive_packets
node_network_transmit_bytes_totalcounterdeviceevery nodeNetwork device statistic transmit_bytes
node_network_transmit_errs_totalcounterdeviceevery nodeNetwork device statistic transmit_errs
node_network_transmit_packets_totalcounterdeviceevery nodeNetwork device statistic transmit_packets
node_time_secondsgauge-every nodeSystem time in seconds since epoch (1970)
node_uname_infogaugemachine, nodename, release, sysname, versionevery nodeLabeled system information as provided by the uname system call
shc_backup_duration_secondshistogramtenant, stackevery nodeTime taken to create backups
shc_backups_created_totalcountertenant, stack, statusevery nodeTotal number of backups created
shc_backups_restored_totalcountertenant, stack, statusevery nodeTotal number of backups restored
shc_clusterfs_snapshot_bytesgauge-every nodeStore value bytes copied by this node’s last clusterfs build
shc_clusterfs_snapshot_rowsgauge-every nodeStore rows copied by this node’s last clusterfs build
shc_clusterfs_snapshot_secondsgauge-every nodeTime taken to copy the store for this node’s last clusterfs build
shc_container_block_read_bytes_totalcountertenant, environment, stack, app, node_id, container_id, container_name, serviceevery nodeTotal bytes read from block devices (cumulative)
shc_container_block_write_bytes_totalcountertenant, environment, stack, app, node_id, container_id, container_name, serviceevery nodeTotal bytes written to block devices (cumulative)
shc_container_cpu_limit_millicoresgaugetenant, environment, stack, app, node_id, container_id, container_name, serviceevery nodeContainer CPU limit in millicores (from compose deploy.resources.limits)
shc_container_cpu_usage_coresgaugetenant, environment, stack, app, node_id, container_id, container_name, serviceevery nodeContainer CPU usage in cores (fractional)
shc_container_health_statusgaugetenant, environment, stack, app, node_id, container_id, container_name, serviceevery nodeContainer health status (1=healthy, 0=unhealthy, -1=none)
shc_container_memory_limit_bytesgaugetenant, environment, stack, app, node_id, container_id, container_name, serviceevery nodeContainer memory limit in bytes
shc_container_memory_usage_bytesgaugetenant, environment, stack, app, node_id, container_id, container_name, serviceevery nodeContainer memory usage in bytes
shc_container_memory_utilization_ratiogaugetenant, environment, stack, app, node_id, container_id, container_name, serviceevery nodeContainer memory utilization as ratio of limit (0-1+)
shc_container_network_receive_bytes_totalcountertenant, environment, stack, app, node_id, container_id, container_name, serviceevery nodeTotal bytes received by container (cumulative)
shc_container_network_transmit_bytes_totalcountertenant, environment, stack, app, node_id, container_id, container_name, serviceevery nodeTotal bytes transmitted by container (cumulative)
shc_container_restart_countgaugetenant, environment, stack, app, node_id, container_id, container_name, serviceevery nodeNumber of container restarts
shc_container_runninggaugetenant, environment, stack, app, node_id, container_id, container_name, serviceevery nodeContainer running state (1=running, 0=stopped)
shc_container_statusgaugedeployment, service, container, statusevery nodeContainer SHC status (1=current status, check ‘status’ label)
shc_crons_activegauge-leaderNumber of active (non-paused) cron jobs
shc_crons_pausedgauge-leaderNumber of paused cron jobs
shc_deployment_statusgaugedeployment, statusevery nodeDeployment SHC status (1=current status, check ‘status’ label)
shc_environment_cpu_scheduled_millicoresgaugetenant, environmentevery nodeEnvironment CPU scheduled in millicores (sum of stack_name declared limits)
shc_environment_cpu_usage_millicoresgaugetenant, environmentevery nodeEnvironment CPU usage in millicores (sum of stacks)
shc_environment_disk_usage_bytesgaugetenant, environmentevery nodeEnvironment disk usage in bytes (sum of stacks)
shc_environment_memory_limit_bytesgaugetenant, environmentevery nodeEnvironment memory limit in bytes (sum of stack_name cgroup limits)
shc_environment_memory_scheduled_bytesgaugetenant, environmentevery nodeEnvironment memory scheduled in bytes (sum of stack_name declared limits)
shc_environment_memory_usage_bytesgaugetenant, environmentevery nodeEnvironment memory usage in bytes (sum of stacks)
shc_event_bus_events_droppedcounter-every nodeEvents dropped due to queue full
shc_event_bus_events_totalcountercode_prefixevery nodeTotal events processed by the event bus
shc_event_bus_handler_errorscounterhandlerevery nodeHandler execution errors
shc_event_bus_queue_depthgauge-every nodeCurrent number of events in the queue
shc_event_bus_runninggauge-every nodeWhether the event bus is running (1=yes, 0=no)
shc_node_containers_runninggaugenode_id, address, roleevery nodeNumber of running containers on node
shc_node_containers_totalgaugenode_id, address, roleevery nodeTotal number of containers on node
shc_node_cpu_capacity_millicoresgaugenode_id, address, role, runtimeevery nodeTotal CPU capacity on node in millicores (hardware)
shc_node_cpu_coresgaugenode_id, address, roleevery nodeNumber of CPU cores on node
shc_node_cpu_load_15mgaugenode_id, address, roleevery nodeNode CPU load average (15 minutes)
shc_node_cpu_load_1mgaugenode_id, address, roleevery nodeNode CPU load average (1 minute)
shc_node_cpu_load_5mgaugenode_id, address, roleevery nodeNode CPU load average (5 minutes)
shc_node_cpu_schedulable_millicoresgaugenode_id, address, role, runtimeevery nodeCPU available for scheduling in millicores (capacity - scheduled)
shc_node_cpu_schedule_utilization_ratiogaugenode_id, address, role, runtimeevery nodeNode CPU scheduling pressure (scheduled / capacity, 0-1+)
shc_node_cpu_scheduled_millicoresgaugenode_id, address, role, runtimeevery nodeCPU scheduled on node in millicores (sum of compose deploy.resources.limits)
shc_node_cpu_utilization_ratiogaugenode_id, address, roleevery nodeNode CPU utilization as ratio (0-1)
shc_node_deployment_countgaugenode_id, address, role, runtimeevery nodeNumber of deployments scheduled on this node
shc_node_disk_available_bytesgaugenode_id, address, role, mountpoint, device, fstypeevery nodeAvailable disk space in bytes
shc_node_disk_capacity_bytesgaugenode_id, address, role, runtimeevery nodeTotal disk capacity on node in bytes (hardware)
shc_node_disk_total_bytesgaugenode_id, address, role, mountpoint, device, fstypeevery nodeTotal disk space in bytes
shc_node_disk_used_bytesgaugenode_id, address, role, mountpoint, device, fstypeevery nodeUsed disk space in bytes
shc_node_disk_utilization_ratiogaugenode_id, address, role, mountpoint, device, fstypeevery nodeDisk utilization as ratio (0-1)
shc_node_docker_upgaugenode_id, address, roleevery nodeDocker daemon status on node (1=up, 0=down)
shc_node_memory_available_bytesgaugenode_id, address, roleevery nodeAvailable memory on node in bytes
shc_node_memory_capacity_bytesgaugenode_id, address, role, runtimeevery nodeTotal memory capacity on node in bytes (hardware)
shc_node_memory_schedulable_bytesgaugenode_id, address, role, runtimeevery nodeMemory available for scheduling in bytes (capacity - scheduled)
shc_node_memory_schedule_utilization_ratiogaugenode_id, address, role, runtimeevery nodeNode memory scheduling pressure (scheduled / capacity, 0-1+)
shc_node_memory_scheduled_bytesgaugenode_id, address, role, runtimeevery nodeMemory scheduled on node in bytes (sum of compose deploy.resources.limits)
shc_node_memory_total_bytesgaugenode_id, address, roleevery nodeTotal memory on node in bytes
shc_node_memory_used_bytesgaugenode_id, address, roleevery nodeUsed memory on node in bytes
shc_node_memory_utilization_ratiogaugenode_id, address, roleevery nodeNode memory utilization as ratio (0-1)
shc_node_network_receive_bytes_totalcounternode_id, address, role, interfaceevery nodeTotal bytes received on interface (cumulative)
shc_node_network_receive_errors_totalcounternode_id, address, role, interfaceevery nodeTotal receive errors on interface (cumulative)
shc_node_network_receive_packets_totalcounternode_id, address, role, interfaceevery nodeTotal packets received on interface (cumulative)
shc_node_network_transmit_bytes_totalcounternode_id, address, role, interfaceevery nodeTotal bytes transmitted on interface (cumulative)
shc_node_network_transmit_errors_totalcounternode_id, address, role, interfaceevery nodeTotal transmit errors on interface (cumulative)
shc_node_network_transmit_packets_totalcounternode_id, address, role, interfaceevery nodeTotal packets transmitted on interface (cumulative)
shc_node_upgaugenode_id, address, roleevery nodeNode availability status (1=up, 0=down)
shc_objchange_audit_failures_totalcounter-every nodeObject changes written whose audit entry could not be written
shc_pressure_depthgaugequeueevery nodeItems waiting in the queue
shc_pressure_dropped_totalcounterqueueevery nodeItems dropped or refused
shc_pressure_in_per_secondgaugequeueevery nodeItems admitted per second, smoothed
shc_pressure_in_totalcounterqueueevery nodeItems admitted
shc_pressure_levelgaugequeueevery nodePressure level (0=green, 1=amber, 2=red)
shc_pressure_oldest_age_secondsgaugequeueevery nodeAge of the oldest waiting item
shc_pressure_out_per_secondgaugequeueevery nodeItems worked per second, smoothed
shc_pressure_out_totalcounterqueueevery nodeItems worked
shc_pressure_retired_totalcounterqueue, reasonevery nodeItems the owner took out for the reason given, neither worked nor dropped; only a queue that retires reports it
shc_queue_depthgauge-every nodeCurrent queue depth (PENDING + RUNNING tasks)
shc_queue_is_leadergauge-every nodeWhether this node is the queue leader (1=yes, 0=no)
shc_queue_registered_cronsgauge-every nodeNumber of cron jobs registered in the queue
shc_queue_registered_handlersgauge-every nodeNumber of event handlers registered in the queue
shc_queue_rejected_totalcounter-every nodeTotal tasks rejected due to backpressure (queue full)
shc_queue_retries_totalcounter-every nodeTotal task retries scheduled after failure or timeout
shc_queue_runninggauge-every nodeWhether the queue is running (1=yes, 0=no)
shc_queue_running_tasksgauge-every nodeNumber of tasks currently executing in this node’s worker pool
shc_queue_timeouts_totalcounter-every nodeTotal tasks terminated due to max_runtime_seconds
shc_quota_apps_limitgaugetenant, environment, stackevery nodeMaximum number of apps allowed
shc_quota_apps_usedgaugetenant, environment, stackevery nodeNumber of apps in use
shc_quota_cpu_limitgaugetenant, environment, stackevery nodeCPU quota limit in millicores
shc_quota_cpu_usedgaugetenant, environment, stackevery nodeCPU usage in millicores (allocated)
shc_quota_cpu_utilization_ratiogaugetenant, environment, stackevery nodeCPU utilization as ratio of limit (0-1+)
shc_quota_disk_limit_bytesgaugetenant, environment, stackevery nodeDisk quota limit in bytes
shc_quota_disk_used_bytesgaugetenant, environment, stackevery nodeDisk usage in bytes (measured)
shc_quota_disk_utilization_ratiogaugetenant, environment, stackevery nodeDisk utilization as ratio of limit (0-1+)
shc_quota_environments_limitgaugetenant, environment, stackevery nodeMaximum number of environments allowed
shc_quota_environments_usedgaugetenant, environment, stackevery nodeNumber of environments in use
shc_quota_memory_limit_bytesgaugetenant, environment, stackevery nodeMemory quota limit in bytes
shc_quota_memory_used_bytesgaugetenant, environment, stackevery nodeMemory usage in bytes (allocated)
shc_quota_memory_utilization_ratiogaugetenant, environment, stackevery nodeMemory utilization as ratio of limit (0-1+)
shc_quota_over_limitgaugetenant, environment, stackevery nodeIndicates if scope is over any quota limit (1=over, 0=ok)
shc_quota_stacks_limitgaugetenant, environment, stackevery nodeMaximum number of stacks allowed
shc_quota_stacks_usedgaugetenant, environment, stackevery nodeNumber of stacks in use
shc_quota_vdeployments_limitgaugetenant, environment, stackevery nodeMaximum number of vdeployments allowed
shc_quota_vdeployments_usedgaugetenant, environment, stackevery nodeNumber of vdeployments installed
shc_rpc_busy_refusals_totalcounterpeerevery nodeInternal RPC calls refused busy at the per-peer in-flight cap
shc_rpc_in_flightgaugepeerevery nodeInternal RPC calls a peer has running on this node
shc_service_cpu_limit_millicoresgaugetenant, environment, stack, app, service, node_idevery nodeService CPU limit in millicores (from compose deploy.resources.limits)
shc_service_cpu_request_millicoresgaugetenant, environment, stack, app, service, node_idevery nodeService CPU reservation in millicores (from compose deploy.resources.reservations)
shc_service_cpu_usage_millicoresgaugetenant, environment, stack, app, service, node_idevery nodeService CPU usage in millicores (sum of containers)
shc_service_memory_limit_bytesgaugetenant, environment, stack, app, service, node_idevery nodeService memory limit in bytes (sum of container cgroup limits)
shc_service_memory_limit_declared_bytesgaugetenant, environment, stack, app, service, node_idevery nodeService memory limit in bytes (from compose deploy.resources.limits)
shc_service_memory_request_bytesgaugetenant, environment, stack, app, service, node_idevery nodeService memory reservation in bytes (from compose deploy.resources.reservations)
shc_service_memory_usage_bytesgaugetenant, environment, stack, app, service, node_idevery nodeService memory usage in bytes (sum of containers)
shc_service_statusgaugedeployment, service, statusevery nodeService SHC status (1=current status, check ‘status’ label)
shc_sse_broker_channelsgauge-every nodeNumber of active SSE broker channels
shc_sse_broker_events_dropped_totalcounter-every nodeEvents dropped due to slow subscriber queues
shc_sse_broker_events_replayed_totalcounter-every nodeEvents replayed from buffer on reconnection
shc_sse_broker_events_totalcounterevent_typeevery nodeTotal events published through SSE broker
shc_sse_broker_session_duration_secondshistogram-every nodeSSE subscriber session duration in seconds
shc_sse_broker_subscribersgauge-every nodeNumber of active SSE broker subscribers
shc_sse_connectionsgaugeoperation_typeevery nodeNumber of active SSE connections
shc_sse_event_latency_secondshistogramevent_typeevery nodeTime from event creation to SSE emission
shc_sse_events_bufferedgaugeoperation_idevery nodeEvents currently buffered per operation
shc_sse_events_dropped_totalcounteroperation_idevery nodeEvents dropped due to buffer overflow
shc_sse_events_emitted_totalcounterevent_typeevery nodeTotal events emitted to SSE streams
shc_sse_memory_bytesgauge-every nodeEstimated memory usage for SSE buffers
shc_sse_operationsgaugestatusevery nodeNumber of tracked operations
shc_sse_slow_clients_totalcounter-every nodeNumber of slow client detections (gap events emitted)
shc_stack_cpu_limit_millicoresgaugetenant, environment, stack, app, node_id, runtimeevery nodeStack CPU limit in millicores (sum of service limits)
shc_stack_cpu_request_millicoresgaugetenant, environment, stack, app, node_id, runtimeevery nodeStack CPU reservation in millicores (sum of service reservations)
shc_stack_cpu_usage_millicoresgaugetenant, environment, stack, app, node_id, runtimeevery nodeStack CPU usage in millicores (sum of services)
shc_stack_disk_usage_bytesgaugetenant, environment, stack, app, node_id, runtimeevery nodeStack disk usage in bytes (from quota disk measurement)
shc_stack_memory_limit_bytesgaugetenant, environment, stack, app, node_id, runtimeevery nodeStack memory limit in bytes (sum of service cgroup limits)
shc_stack_memory_limit_declared_bytesgaugetenant, environment, stack, app, node_id, runtimeevery nodeStack memory limit in bytes (sum of service limits)
shc_stack_memory_request_bytesgaugetenant, environment, stack, app, node_id, runtimeevery nodeStack memory reservation in bytes (sum of service reservations)
shc_stack_memory_usage_bytesgaugetenant, environment, stack, app, node_id, runtimeevery nodeStack memory usage in bytes (sum of services)
shc_stacks_activegaugetenant, environmentevery nodeNumber of currently active stacks
shc_stacks_deployed_totalcountertenant, app, environmentevery nodeTotal number of stacks deployed
shc_subsystem_stategaugesubsystemevery nodeSubsystem state (1=running, 0.5=degraded, 0=stopped, -1=failed)
shc_tasks_totalgaugestatusleaderNumber of tasks by status
shc_tenant_cpu_scheduled_millicoresgaugetenantevery nodeTenant CPU scheduled in millicores (sum of declared limits)
shc_tenant_cpu_usage_millicoresgaugetenantevery nodeTenant CPU usage in millicores (sum of environments)
shc_tenant_disk_usage_bytesgaugetenantevery nodeTenant disk usage in bytes (sum of environments)
shc_tenant_memory_limit_bytesgaugetenantevery nodeTenant memory limit in bytes (sum of environment cgroup limits)
shc_tenant_memory_scheduled_bytesgaugetenantevery nodeTenant memory scheduled in bytes (sum of declared limits)
shc_tenant_memory_usage_bytesgaugetenantevery nodeTenant memory usage in bytes (sum of environments)
shc_transfer_activegaugeside, classevery nodeTransfers running on this node: serving (side=serve), pulling (side=fetch) or staging (side=stage)
shc_transfer_busy_refusals_totalcounterside, classevery nodeTransfers refused busy: queue full or no slot before the caller’s deadline
shc_transfer_bytes_per_secondgaugedirectionevery nodeTransfer throughput over the last tick, received (direction=in) or served (direction=out)
shc_transfer_bytes_totalcounterdirectionevery nodeTransfer bytes received (direction=in) or served (direction=out)
shc_transfer_hash_failures_totalcounter-every nodePulls whose sha256 did not match the holder’s
shc_transfer_oldest_wait_secondsgaugeside, classevery nodeAge of the oldest transfer waiting for a slot
shc_transfer_peer_activegaugeside, class, peerevery nodeTransfers running on this node per peer
shc_transfer_quota_bytesgauge-every nodeThis node’s staging quota
shc_transfer_quota_refusals_totalcounter-every nodeStaging or receiving refused by the quota or the disk’s free space
shc_transfer_staged_bytesgauge-every nodeBytes staged on this node for peers to pull
shc_transfer_sweeps_totalcounter-every nodeExpired staged handles and partial pulls swept
shc_transfer_waitersgaugeside, classevery nodeTransfers waiting for a slot
shc_vault_operations_totalcountertenant, operation, statusevery nodeTotal vault operations
shc_vault_secrets_totalgaugetenant, scopeevery nodeTotal number of secrets in vault
shc_workers_leadergauge-leaderNumber of workers with leader status
shc_workers_totalgauge-leaderNumber of registered workers