Events API
The events module exposes two SSE streams plus an internal peer-broadcast
ingest endpoint. SHC adds an operator gauge and an optional manual
cleanup on the same event table. The streams are tenant-scoped,
support resume via Last-Event-ID, and emit SSE frames with monotonic
per-node id: values backed by the event table.
| Method | Path | Purpose |
|---|---|---|
GET | /api/method/shc.events.stream | Tenant-scoped SSE stream of all events |
GET | /api/method/shc.events.errors.stream | Errors-only variant of the stream |
GET | /api/method/shc.events.pressure | How close event is to overflowing (events:read) |
POST | /api/method/shc.events.prune | Run non-audit retention now (tenant:manage) |
GET /api/method/shc.events.stream
Section titled “GET /api/method/shc.events.stream”Live SSE stream of events for one tenant. Heartbeats every 30s; one SSE frame per event.
Tenant scoping and auth
Section titled “Tenant scoping and auth”The stream requires an authenticated caller with events:read. It reads
the tenant the request names, in ?tenant_name= or the X-SHC-Tenant
scope, and otherwise the authenticated user’s own tenant; a caller naming
a tenant it may not read is refused X403904NOTENT, and one that resolves
no tenant at all is refused X400976TNTNON. Two widenings exist:
- the UDS local user who names no tenant reads every tenant (FZ-7922: one who names a tenant reads that tenant’s events);
- a platform admin may pass
?all_tenants=1.
Header-less clients (a browser EventSource cannot set Authorization)
authenticate with ?token=<jwt>: the daemon accepts the query-param
token specifically for SSE/WebSocket clients and flags it X200067QRYTOK.
Query parameters
Section titled “Query parameters”| Param | Required | Description |
|---|---|---|
token | No | Bearer JWT for header-less clients (EventSource). Authorization: Bearer wins when both are present. |
all_tenants | No | 1 widens to every tenant (platform admins only). |
stack_name | No | Restricts to one stack. |
node_id | No | Restricts to events originated by one node. |
code | No | Restricts to one event code. |
type | No | Comma-separated whitelist of classified types (audit, docker, error, event). |
last_event_id | No | Fallback resume cursor for clients that can’t set headers. |
Headers
Section titled “Headers”| Header | Direction | Description |
|---|---|---|
Authorization: Bearer <jwt> | Request | Keycloak-issued token; the tenant is read from it. |
Last-Event-ID | Request | RFC EventSource resume cursor. See Replay below. |
Content-Type: text/event-stream | Response | Always set. |
Cache-Control: no-cache | Response | Always set. |
Frame shape
Section titled “Frame shape”id: 47event: dockerdata: {"code":"D100100CTRSTR","message":"container started",...,"sequence":47,"event_type":"docker"}The id: field carries the originating node’s event row id,
monotonic per node. Clients echo it on the next reconnect via
Last-Event-ID; the server replays anything newer.
id: is present only on DURABLE events. Ephemeral events (the raw
docker container/exec/health observations; see the durability column in
the event reference) never enter event, so they
have no row id and no resume cursor. On an idle estate they are the
overwhelming majority of frames: a 95-second capture on one production
node took 3,173 event frames, of which 3,172 carried no id:.
Synthesising an id for them would be worse than the omission: the client
would ship a number event never issued as its resume point, and
the replay would silently return nothing. So the two facts have two
fields:
| Field | Meaning | Present on |
|---|---|---|
id: / sequence | where to RESUME from | durable events only |
local_sequence | per-node bus counter: did I MISS any? | every event |
local_sequence is stamped as the event enters the daemon’s in-process
bus, before the non-blocking enqueue, so an event dropped for
capacity leaves a visible hole. Consecutive frames on one connection
carry consecutive values; a jump is an event this node dropped.
Heartbeat frames
Section titled “Heartbeat frames”event: heartbeatdata: {"ts":"2026-08-18T04:00:00Z","verdict":"unmatched", "reason":"stack \"ghost\" does not exist in tenant corp", "scope":{"tenant":"corp","all_tenants":false,"stack":"ghost"}, "bus":{"observed":3102,"matched":3102,"delivered":3102,"dropped":0}, "stream":{"sent":0,"rejected":3102,"last_sequence":0,"uptime_seconds":94}, "cluster":{"source_hostname":"compose01","source_node_id":"node-82356661f66b", "cross_node_delivery":false,"peers":0}}One is sent IMMEDIATELY on connect and then every 30 seconds. ts keeps
its original meaning and position for pre-existing clients; everything
else is additive.
Why a heartbeat carries diagnostics. A stream that delivers nothing
looks identical whether the estate is idle or the subscription is scoped
to something that never matches. That ambiguity is not hypothetical: a
console authenticated against realm corp sat on a 33-events/second feed
showing a green “Live” chip and an empty list for weeks, because every
frame carried tenant="default" and the tenant filter is exact-match. It
received exactly the heartbeats a healthy idle feed sends.
| Field | Meaning |
|---|---|
verdict | live: events matched and were sent. unmatched: nothing was sent and the scope can never match. The stack does not exist in the tenant (in any tenant for an all-tenants stream), the caller’s stack grants exclude it, or type names no event class. An empty screen is a DEFECT in the scope. idle: nothing was sent and the scope can match, so an empty screen is correct however busy the bus is. The counters below are evidence and never decide the verdict (SHC-956): a stack-scoped stream on a busy tenant refuses every other stack’s frame, and that says nothing about the stack. A stack lookup that fails reads as idle. |
reason | Why the scope can never match. Present only with unmatched. |
scope | The scope the SERVER applied. Echoed because the caller does not choose most of it: the tenant comes off the bearer token and the stack set off the caller’s role grants. |
bus.observed | Events the bus offered this subscription before its filter ran. It separates a quiet bus from a busy one; it does not decide the verdict. |
bus.matched | The subset that passed the tenant/stack filter. |
bus.dropped | Matched events lost because this subscriber’s queue was full: a slow consumer, not a scope problem. |
stream.rejected | Events that passed the subscription filter and were then refused by the per-stream query (type, errors-only). Non-zero with sent == 0 means the QUERY is too narrow, which is a different repair from a wrong tenant. |
cluster.source_hostname | The serving daemon’s hostname, the same value stamped on every event row it publishes. |
cluster.source_node_id | The serving node’s SHC node id, the id-space a stack’s node/nodes use. Compare THIS against a stack’s nodes, never source_hostname: a hostname never equals a node id (SHC-569). Omitted when the id cannot be resolved, and never filled from the hostname. |
cluster.cross_node_delivery | false means no peer-discovery seam is wired: this stream carries only events published on the node serving it and omits every peer’s. See shc doctor run --category events. |
shc events --follow prints a one-line hint to stderr (once per
verdict, never into the event stream) when a heartbeat reports
unmatched, a non-zero drop count, or a single-node view.
Replay
Section titled “Replay”When the request carries Last-Event-ID: N (or ?last_event_id=N),
the handler:
- Subscribes to the live channel first so events emitted during step 2 are queued.
- Reads the current
MAX(id)fromeventfor the tenant: the replay watermarkW. - Streams rows where
id > N AND id <= W, in id order, as SSE frames (capped at 1000 to avoid pinning the connection on a long-gone client). - Switches to forwarding live subscription events, dropping any with
Sequence <= Wso a dupe between replay and live is impossible.
Limitations:
- The sequence space is per-node, not cluster-wide. A multi-node cluster’s replay covers only events the local node has persisted; cross-node events that arrived via the peer-broadcast path are included once the receive path persists them (see below); events the node never observed are not replayable.
- The 1000-frame cap means a client offline long enough to accumulate more than 1000 missed events will see a gap. Reconnect aggressively if your workload generates events faster than your clients consume them.
Event classification
Section titled “Event classification”event: line is derived from the Event:
| Source | Classified as |
|---|---|
IsAudit == true or code prefix A | audit |
level == "error" or code prefix E | error |
Code prefix D | docker |
| Anything else | event |
GET /api/method/shc.events.errors.stream
Section titled “GET /api/method/shc.events.errors.stream”Same wire shape as the main stream but filtered to level == "error"
events only. Useful for ops dashboards that only care about failure
signals.
The type= query parameter is ignored on this endpoint (the
errors-only filter overrides). All other query parameters work
identically.
GET /api/method/shc.events.pressure
Section titled “GET /api/method/shc.events.pressure”Operator gauge for event. Gated events:read. Envelope is
{"message": …}.
| Field | When present |
|---|---|
rows | Current table row count (audit included) |
max_rows, percent_full | Only when maintenance.event_max_rows is > 0 |
row_cap | "capped" while the store is at its cap: at or over it, or within 1% under it after a cap trim in the last three hours. "under" while a cap is set and the store sits below that. "unbounded" when maintenance.event_max_rows is 0 |
retention_days | Effective age policy (maintenance.event_retention_days, default 14) |
oldest_age_days, prune_in_days | Oldest non-audit row versus retention. Omitted on an empty table |
pending_broadcast | Non-audit rows still pending_broadcast=1 |
stuck_pending | Those pending rows past maintenance.event_broadcast_grace_days (default 1). The X503114EVBRGU class |
broadcast_grace_days | The grace window |
last_prune | Last successful shc.events.prune in this process. Omitted until one runs. The hourly cleanup cron does not persist a report this endpoint can read |
can_prune | Caller holds tenant:manage (or is the UDS local user / platform admin) |
severity | ok, warning (≥80% full, prune in ≤1 day, or any stuck broadcast), or critical (≥95% full or already due) |
maintenance.disk_auto_prune_threshold is not in this payload: it trips
backup-archive prune, not event.
Audit retention stays off here. The CE cleanup cron may now default the
is_audit half of event to 90 days; this gauge’s prune-in clock
ignores audit rows so an ancient compliance mirror cannot lie about when
operational cleanup fires.
POST /api/method/shc.events.prune
Section titled “POST /api/method/shc.events.prune”Optional one-tap cleanup. Gated tenant:manage. The hourly leader-only
cleanup cron remains the authority. This verb runs the same non-audit
predicates (retention_days, max_rows, short/long tiers, broadcast
grace, pending TTL) and forces audit knobs off (AuditRetentionDays=0,
AuditMaxRows=0, PurgeAuditRows=false). It does not call
shc.db.purge and does not delete the audit table.
Response envelope {"message": {at, aged, capped, stuck_pending, total}}.
stuck_pending is the grace backstop (ordinary vs backstop); total is
their sum. A successful run is what last_prune on the pressure GET
then reports.
Internal RPC function events.deliver
Section titled “Internal RPC function events.deliver”Cross-node peer-broadcast receiver, served through
POST /internal/rpc/events.deliver on the internal listener (not the
public API). The NodeBroadcaster calls it on every reachable peer when
this node publishes an event. Source:
modules/events/rpc.go.
Arguments
Section titled “Arguments”| Argument | Required | Description |
|---|---|---|
source_hostname | Yes | Originating node. Missing is refused with X400000BADREQ. |
event | Yes | One Event.ToDict() envelope, as a JSON string. |
The envelope mirrors the JSON data: payload of the SSE frame format
(see above): same keys, same types.
Behavior
Section titled “Behavior”- If
source_hostnameequals this node’s own SourceHostname the event is dropped and the call answers{"delivered": false}(loop prevention per B12 Bug 5). - The envelope is decoded into an
Event. Nothing is written toevent: on a multi-node cluster the table is raft-replicated, so the originating node’s row is already this node’s row. - Local subscribers receive the event via the in-process bus, with the
originator’s
sequenceas the resume cursor.
Answers
Section titled “Answers”| Answer | Meaning |
|---|---|
{"delivered": true} | Delivered to local subscribers. |
{"delivered": false} | Self-loop, dropped without action. |
X400000BADREQ | Missing source_hostname, or an event that is not a JSON string, not JSON, or has no code. |
X500007EVTBUS | Bus not running. |
The broadcaster’s retry queue applies backoff to an error; either answer is treated as acked.