Skip to content
SHC Docs

Events API

The events module exposes two SSE streams plus an internal peer-broadcast ingest endpoint. SHC adds an operator gauge and an optional manual cleanup on the same event table. The streams are tenant-scoped, support resume via Last-Event-ID, and emit SSE frames with monotonic per-node id: values backed by the event table.

MethodPathPurpose
GET/api/method/shc.events.streamTenant-scoped SSE stream of all events
GET/api/method/shc.events.errors.streamErrors-only variant of the stream
GET/api/method/shc.events.pressureHow close event is to overflowing (events:read)
POST/api/method/shc.events.pruneRun non-audit retention now (tenant:manage)

Live SSE stream of events for one tenant. Heartbeats every 30s; one SSE frame per event.

The stream requires an authenticated caller with events:read. It reads the tenant the request names, in ?tenant_name= or the X-SHC-Tenant scope, and otherwise the authenticated user’s own tenant; a caller naming a tenant it may not read is refused X403904NOTENT, and one that resolves no tenant at all is refused X400976TNTNON. Two widenings exist:

  • the UDS local user who names no tenant reads every tenant (FZ-7922: one who names a tenant reads that tenant’s events);
  • a platform admin may pass ?all_tenants=1.

Header-less clients (a browser EventSource cannot set Authorization) authenticate with ?token=<jwt>: the daemon accepts the query-param token specifically for SSE/WebSocket clients and flags it X200067QRYTOK.

ParamRequiredDescription
tokenNoBearer JWT for header-less clients (EventSource). Authorization: Bearer wins when both are present.
all_tenantsNo1 widens to every tenant (platform admins only).
stack_nameNoRestricts to one stack.
node_idNoRestricts to events originated by one node.
codeNoRestricts to one event code.
typeNoComma-separated whitelist of classified types (audit, docker, error, event).
last_event_idNoFallback resume cursor for clients that can’t set headers.
HeaderDirectionDescription
Authorization: Bearer <jwt>RequestKeycloak-issued token; the tenant is read from it.
Last-Event-IDRequestRFC EventSource resume cursor. See Replay below.
Content-Type: text/event-streamResponseAlways set.
Cache-Control: no-cacheResponseAlways set.
id: 47
event: docker
data: {"code":"D100100CTRSTR","message":"container started",...,"sequence":47,"event_type":"docker"}

The id: field carries the originating node’s event row id, monotonic per node. Clients echo it on the next reconnect via Last-Event-ID; the server replays anything newer.

id: is present only on DURABLE events. Ephemeral events (the raw docker container/exec/health observations; see the durability column in the event reference) never enter event, so they have no row id and no resume cursor. On an idle estate they are the overwhelming majority of frames: a 95-second capture on one production node took 3,173 event frames, of which 3,172 carried no id:.

Synthesising an id for them would be worse than the omission: the client would ship a number event never issued as its resume point, and the replay would silently return nothing. So the two facts have two fields:

FieldMeaningPresent on
id: / sequencewhere to RESUME fromdurable events only
local_sequenceper-node bus counter: did I MISS any?every event

local_sequence is stamped as the event enters the daemon’s in-process bus, before the non-blocking enqueue, so an event dropped for capacity leaves a visible hole. Consecutive frames on one connection carry consecutive values; a jump is an event this node dropped.

event: heartbeat
data: {"ts":"2026-08-18T04:00:00Z","verdict":"unmatched",
"reason":"stack \"ghost\" does not exist in tenant corp",
"scope":{"tenant":"corp","all_tenants":false,"stack":"ghost"},
"bus":{"observed":3102,"matched":3102,"delivered":3102,"dropped":0},
"stream":{"sent":0,"rejected":3102,"last_sequence":0,"uptime_seconds":94},
"cluster":{"source_hostname":"compose01","source_node_id":"node-82356661f66b",
"cross_node_delivery":false,"peers":0}}

One is sent IMMEDIATELY on connect and then every 30 seconds. ts keeps its original meaning and position for pre-existing clients; everything else is additive.

Why a heartbeat carries diagnostics. A stream that delivers nothing looks identical whether the estate is idle or the subscription is scoped to something that never matches. That ambiguity is not hypothetical: a console authenticated against realm corp sat on a 33-events/second feed showing a green “Live” chip and an empty list for weeks, because every frame carried tenant="default" and the tenant filter is exact-match. It received exactly the heartbeats a healthy idle feed sends.

FieldMeaning
verdictlive: events matched and were sent. unmatched: nothing was sent and the scope can never match. The stack does not exist in the tenant (in any tenant for an all-tenants stream), the caller’s stack grants exclude it, or type names no event class. An empty screen is a DEFECT in the scope. idle: nothing was sent and the scope can match, so an empty screen is correct however busy the bus is. The counters below are evidence and never decide the verdict (SHC-956): a stack-scoped stream on a busy tenant refuses every other stack’s frame, and that says nothing about the stack. A stack lookup that fails reads as idle.
reasonWhy the scope can never match. Present only with unmatched.
scopeThe scope the SERVER applied. Echoed because the caller does not choose most of it: the tenant comes off the bearer token and the stack set off the caller’s role grants.
bus.observedEvents the bus offered this subscription before its filter ran. It separates a quiet bus from a busy one; it does not decide the verdict.
bus.matchedThe subset that passed the tenant/stack filter.
bus.droppedMatched events lost because this subscriber’s queue was full: a slow consumer, not a scope problem.
stream.rejectedEvents that passed the subscription filter and were then refused by the per-stream query (type, errors-only). Non-zero with sent == 0 means the QUERY is too narrow, which is a different repair from a wrong tenant.
cluster.source_hostnameThe serving daemon’s hostname, the same value stamped on every event row it publishes.
cluster.source_node_idThe serving node’s SHC node id, the id-space a stack’s node/nodes use. Compare THIS against a stack’s nodes, never source_hostname: a hostname never equals a node id (SHC-569). Omitted when the id cannot be resolved, and never filled from the hostname.
cluster.cross_node_deliveryfalse means no peer-discovery seam is wired: this stream carries only events published on the node serving it and omits every peer’s. See shc doctor run --category events.

shc events --follow prints a one-line hint to stderr (once per verdict, never into the event stream) when a heartbeat reports unmatched, a non-zero drop count, or a single-node view.

When the request carries Last-Event-ID: N (or ?last_event_id=N), the handler:

  1. Subscribes to the live channel first so events emitted during step 2 are queued.
  2. Reads the current MAX(id) from event for the tenant: the replay watermark W.
  3. Streams rows where id > N AND id <= W, in id order, as SSE frames (capped at 1000 to avoid pinning the connection on a long-gone client).
  4. Switches to forwarding live subscription events, dropping any with Sequence <= W so a dupe between replay and live is impossible.

Limitations:

  • The sequence space is per-node, not cluster-wide. A multi-node cluster’s replay covers only events the local node has persisted; cross-node events that arrived via the peer-broadcast path are included once the receive path persists them (see below); events the node never observed are not replayable.
  • The 1000-frame cap means a client offline long enough to accumulate more than 1000 missed events will see a gap. Reconnect aggressively if your workload generates events faster than your clients consume them.

event: line is derived from the Event:

SourceClassified as
IsAudit == true or code prefix Aaudit
level == "error" or code prefix Eerror
Code prefix Ddocker
Anything elseevent

Same wire shape as the main stream but filtered to level == "error" events only. Useful for ops dashboards that only care about failure signals.

The type= query parameter is ignored on this endpoint (the errors-only filter overrides). All other query parameters work identically.


Operator gauge for event. Gated events:read. Envelope is {"message": …}.

FieldWhen present
rowsCurrent table row count (audit included)
max_rows, percent_fullOnly when maintenance.event_max_rows is > 0
row_cap"capped" while the store is at its cap: at or over it, or within 1% under it after a cap trim in the last three hours. "under" while a cap is set and the store sits below that. "unbounded" when maintenance.event_max_rows is 0
retention_daysEffective age policy (maintenance.event_retention_days, default 14)
oldest_age_days, prune_in_daysOldest non-audit row versus retention. Omitted on an empty table
pending_broadcastNon-audit rows still pending_broadcast=1
stuck_pendingThose pending rows past maintenance.event_broadcast_grace_days (default 1). The X503114EVBRGU class
broadcast_grace_daysThe grace window
last_pruneLast successful shc.events.prune in this process. Omitted until one runs. The hourly cleanup cron does not persist a report this endpoint can read
can_pruneCaller holds tenant:manage (or is the UDS local user / platform admin)
severityok, warning (≥80% full, prune in ≤1 day, or any stuck broadcast), or critical (≥95% full or already due)

maintenance.disk_auto_prune_threshold is not in this payload: it trips backup-archive prune, not event.

Audit retention stays off here. The CE cleanup cron may now default the is_audit half of event to 90 days; this gauge’s prune-in clock ignores audit rows so an ancient compliance mirror cannot lie about when operational cleanup fires.


Optional one-tap cleanup. Gated tenant:manage. The hourly leader-only cleanup cron remains the authority. This verb runs the same non-audit predicates (retention_days, max_rows, short/long tiers, broadcast grace, pending TTL) and forces audit knobs off (AuditRetentionDays=0, AuditMaxRows=0, PurgeAuditRows=false). It does not call shc.db.purge and does not delete the audit table.

Response envelope {"message": {at, aged, capped, stuck_pending, total}}. stuck_pending is the grace backstop (ordinary vs backstop); total is their sum. A successful run is what last_prune on the pressure GET then reports.


Cross-node peer-broadcast receiver, served through POST /internal/rpc/events.deliver on the internal listener (not the public API). The NodeBroadcaster calls it on every reachable peer when this node publishes an event. Source: modules/events/rpc.go.

ArgumentRequiredDescription
source_hostnameYesOriginating node. Missing is refused with X400000BADREQ.
eventYesOne Event.ToDict() envelope, as a JSON string.

The envelope mirrors the JSON data: payload of the SSE frame format (see above): same keys, same types.

  1. If source_hostname equals this node’s own SourceHostname the event is dropped and the call answers {"delivered": false} (loop prevention per B12 Bug 5).
  2. The envelope is decoded into an Event. Nothing is written to event: on a multi-node cluster the table is raft-replicated, so the originating node’s row is already this node’s row.
  3. Local subscribers receive the event via the in-process bus, with the originator’s sequence as the resume cursor.
AnswerMeaning
{"delivered": true}Delivered to local subscribers.
{"delivered": false}Self-loop, dropped without action.
X400000BADREQMissing source_hostname, or an event that is not a JSON string, not JSON, or has no code.
X500007EVTBUSBus not running.

The broadcaster’s retry queue applies backoff to an error; either answer is treated as acked.