Unified event bus
Layer 2. Core. Core
The Unified Event Bus is the central event routing system in SHC. It receives all events from log.Event(), log.Error(), and log.Warn() and routes them to handlers, cross-node broadcast, and SSE streaming.
Overview
Section titled “Overview”The Problem
Section titled “The Problem”Previously, events flowed through multiple disconnected paths:
log.Event()→ DB polling → task handlers- Docker events → SSE broadcast → clients
- Separate notification dispatch
This fragmentation caused:
- Inconsistent event handling
- Complex debugging
- No unified tracing
- Different tracking mechanisms for different dispatch modes
The Solution
Section titled “The Solution”A single in-memory event bus that:
- Receives all events via
log.Event(),log.Error(),log.Warn() - Routes to local handlers based on dispatch mode
- Broadcasts to other nodes via HTTP POST
- Publishes to SSE channels for client streaming
Architecture
Section titled “Architecture”flowchart TB LOG["log.Event() / log.Error() / log.Warn()"] --> BUS["EventBus (in-memory)<br/>event channel · consumer goroutine · handler registry"] BUS --> LH["Local Handlers"] BUS --> XN["Cross-Node<br/>RPC events.deliver"] BUS --> SSE["SSE Broadcast<br/>(progress, audit, etc.)"]Event Structure
Section titled “Event Structure”Events are represented by the Event struct:
type Event struct { // Identity Code string `json:"code"` // A200100STKINS, X500907STKFAL Message string `json:"message"` // Rendered human-readable message Payload map[string]any `json:"payload"` // Event-specific data
// Tracing TraceID string `json:"trace_id"` // 32 hex chars SpanID string `json:"span_id"` // 16 hex chars ParentSpanID string `json:"parent_span_id"` // Optional
// Scope TenantName string `json:"tenant"` // Default: "default" StackName string `json:"stack"` // Optional Environment string `json:"environment"` // Optional Actor string `json:"actor"` // Default: "shc"
// Metadata Timestamp time.Time `json:"timestamp"` SourceHostname string `json:"source_hostname"` Level Level `json:"level"` // info, warning or error
// Flags IsAudit bool `json:"is_audit"` // A-prefix}Wire format note: when serialized for the RPC bus or SSE, the scope fields are emitted as
tenant/stack(without the_namesuffix) for backward compatibility with external consumers. The Go struct usesTenantName/StackNameinternally to matchctxand the rest of the codebase.
Event Code Format
Section titled “Event Code Format”Event codes follow a specific format:
| Prefix | Meaning | Example |
|---|---|---|
V | Verbose/info event | V200300CTRSTR - container started |
X | Coded error or warning fanned onto the bus | X500907STKFAL - deployment operation failed |
A | Audit event | A200100STKINS - deployment installed |
Format: {prefix}{http_status}{sequence}{mnemonic}
- Prefix: 1 char (V/X/A)
- HTTP status: 3 digits
- Sequence: 3 digits
- Mnemonic: 6+ chars
There is no dedicated warning prefix: warnings ride X2xxxxx codes
published through the typed logger at level warning.
Dispatch Modes
Section titled “Dispatch Modes”Events can trigger handlers in three dispatch modes:
SINGLE
Section titled “SINGLE”Only one node executes the handler. Used for tasks that must run exactly once.
func init() { events.Register("A200100STKINS", events.DispatchSingle, handleInstall)}
func handleInstall(ctx context.Context, e *events.Event) error { // Only one node executes this return nil}- Task is enqueued to database
- First node to claim the task executes it
- Status tracked in
node_executionsJSONB field
BROADCAST
Section titled “BROADCAST”All nodes execute the handler. Used for cache invalidation, local state updates.
func init() { events.Register("V200100CFGCHG", events.DispatchBroadcast, handleConfigChange)}
func handleConfigChange(ctx context.Context, e *events.Event) error { // All nodes execute this reloadLocalConfig() return nil}- Executed immediately on receiving node
- All nodes track execution in
node_executions - Task is “complete” when all nodes finish
LEADER
Section titled “LEADER”Only the leader node executes the handler. Used for cluster-wide coordination.
func init() { events.Register("V200401BKPCMP", events.DispatchLeader, handleBackup)}
func handleBackup(ctx context.Context, e *events.Event) error { // Only leader executes this return nil}- Only executes if current node is leader
- Enqueued to database for tracking
- Falls back to SINGLE if no leader
Event Flow
Section titled “Event Flow”1. Event Publication
Section titled “1. Event Publication”Events are published via log.Event():
import "gitlab.com/bitspur/selfhosted-cloud/ce/core/log"
log.Event(ctx, events.A200100STKINS, map[string]any{ "stack_name": "postgres", "version": "16.2",})This:
- Gets current context from
ctx - Creates an
Eventwith tracing info - Publishes to the event bus channel
2. Event Processing
Section titled “2. Event Processing”The event bus consumer goroutine:
- Receives events from the channel
- Routes to registered handlers
- Broadcasts to other nodes (if local origin)
- Publishes to SSE channels
3. Handler Execution
Section titled “3. Handler Execution”For each registered handler:
// BROADCAST: Execute immediately in goroutinego executeHandler(handler, event, kwargs)
// LEADER: Execute if leaderif isLeader() { enqueueTask(handler, event, kwargs, DispatchLeader)}
// SINGLE: Enqueue for claim-based executionenqueueTask(handler, event, kwargs, DispatchSingle)Handlers receive context with event details:
func handleInstall(ctx context.Context, e *events.Event) error { slog.InfoContext(ctx, "Handling event", "code", e.Code) stack := e.Payload["stack"].(string) return nil}Cross-Node Broadcasting
Section titled “Cross-Node Broadcasting”Events are broadcast to other nodes with the events.deliver internal RPC function:
sequenceDiagram participant A as Node A (origin) participant B as Node B participant C as Node C Note over A: Event created A->>B: RPC events.deliver A->>C: RPC events.deliver Note over B,C: Event received<br/>Published to local bus<br/>Handlers execute<br/>NOT re-broadcastNodeBroadcaster
Section titled “NodeBroadcaster”The NodeBroadcaster (modules/events/broadcaster.go)
handles cross-node delivery. Per B12 Bug 2 FIX, delivery is no
longer fire-and-forget: each peer has a per-peer in-memory retry
queue with bounded exponential backoff (5 attempts, base 200ms,
jitter, 30s wall budget). After exhaustion the
shc_event_bus_peer_failures_total{peer=…} counter increments.
The event travels as Event.ToDict(), the same JSON envelope the SSE
stream’s data: payload uses, including the sequence field once
the originating node persists the row. It is passed as one JSON string
in the event argument, next to source_hostname. The RPC layer reads a
one-key $blob or $transfer map as a binary reference, and an event
payload is free to hold one, so the event is never sent as a nested
object.
events.deliver
Section titled “events.deliver”Other nodes receive events through events.deliver, served by the one
internal RPC dispatch (POST /internal/rpc/events.deliver) on the
internal listener only. The function lives in
modules/events/rpc.go
and enforces the loop-prevention contract from B12 Bug 5:
source_hostnameis required. A delivery without it is refused withX400000BADREQ. The broadcaster always sets it.- Self-loop drop. If
source_hostnameequals this node’s ownSourceHostname, the call answers{delivered: false}without touching the bus, so the sender does not retry and a wrong peer address cannot amplify into a storm. - Deliver, never persist. The envelope decodes into an
Eventand fans to in-process subscribers throughDeliverPeer. Nothing is written toevent: on a multi-node cluster that table is the raft-replicated database, so the originator’s row is already this node’s row, and its id stays the SSE resume cursor.
Answers
Section titled “Answers”| Answer | Meaning |
|---|---|
{delivered: true} | Delivered to local subs. Broadcaster acks. |
{delivered: false} | Self-loop dropped. Broadcaster acks (no retry). |
X400000BADREQ | Missing source_hostname or malformed event. Counts as a failed attempt. |
X500007EVTBUS | Bus not running. Broadcaster retries with backoff. |
SSE Progress Streaming
Section titled “SSE Progress Streaming”Events are published to SSE channels for client streaming:
Channel Format
Section titled “Channel Format”| Channel | Description |
|---|---|
operation:{tenant}:{operation_id} | Operation progress (step, status, complete, error events) |
events:{tenant} | All events (docker, audit, etc.) for a tenant |
errors:{tenant} | Error events for a tenant |
Client Connection Flow
Section titled “Client Connection Flow”sequenceDiagram participant Client participant Server Client->>Server: POST /api/method/shc.backup.create_async Server-->>Client: 202 Accepted<br/>traceparent: 00-abc123...-01<br/>{"operation_id": "abc123..."} Client->>Server: GET /api/method/shc.operation.stream/abc123... Server-->>Client: SSE Events<br/>event: step<br/>data: {"step": "validating"}SSE Event Types (operation:{tenant}:{operation_id} channel)
Section titled “SSE Event Types (operation:{tenant}:{operation_id} channel)”| Event Type | Description |
|---|---|
step | Progress update with step name, percentage, and message |
container_status | Container state changes |
service_status | Service replica counts and status |
stack_status | Overall stack status changes |
complete | Operation completed successfully |
error | Operation failed |
SSE Event Data Format
Section titled “SSE Event Data Format”{ "code": "A200100STKINS", "message": "Stack installed successfully", "payload": {"stack_name": "postgres", "version": "16.2"}, "timestamp": "2026-02-04T12:00:00.000Z", "node": "node-1"}Task Tracking
Section titled “Task Tracking”Tasks are tracked in the database with a node_executions JSONB field:
type NodeExecution struct { NodeID string `json:"node_id"` Status NodeExecutionStatus `json:"status"` // PENDING, RUNNING, COMPLETED, FAILED, SKIPPED StartedAt *time.Time `json:"started_at"` CompletedAt *time.Time `json:"completed_at"` Error string `json:"error,omitempty"`}Status Derivation
Section titled “Status Derivation”Overall task status is derived from node executions:
| Dispatch Mode | Status Logic |
|---|---|
| SINGLE | Status = single execution’s status |
| LEADER | Status = leader execution’s status |
| BROADCAST | ALL must complete, ANY failure = failed |
Handler Registration
Section titled “Handler Registration”Handlers are registered via the events.Register function:
import "gitlab.com/bitspur/selfhosted-cloud/ce/modules/events"
func init() { events.Register("A200100STKINS", events.DispatchSingle, handleStackInstalled)}
func handleStackInstalled(ctx context.Context, e *events.Event) error { stack := e.Payload["stack"].(string) return notifyUsers(ctx, stack)}Multiple handlers can be registered for the same event code.
Event Bus Lifecycle
Section titled “Event Bus Lifecycle”The event bus is managed as a service:
// In server startupmanager.Register(NewEventBusComponent(), 5)
// EventBusComponent delegates to eventBus.Start() / Stop()Start Priority
Section titled “Start Priority”The event bus starts early (priority 5) so other components can publish events during their startup.
Wiring SSE and Broadcaster
Section titled “Wiring SSE and Broadcaster”After startup, the SSE broker and node broadcaster are wired into the event bus:
// In server.goeventBus.SetSSEBroker(sseBroker)eventBus.SetNodeBroadcaster(nodeBroadcaster)Error Handling
Section titled “Error Handling”Queue Full
Section titled “Queue Full”If the event channel buffer is full (default: 10,000 events), new events are dropped:
select {case bus.queue <- event: // sentdefault: slog.Debug("event bus queue full, dropping event", "code", event.Code)}Design decision: Fire-and-forget. Events are ephemeral. Critical state uses polling.
Handler Exceptions
Section titled “Handler Exceptions”A handler is isolated twice over: a returned error and a panic are
both contained, and neither takes the bus down or skips the remaining
handlers for that code. Both are handed to the service’s
OnHandlerError seam, which the daemon wires to the log:
func (s *Service) invokeTaskHandler(ctx context.Context, reg TaskRegistration, e *Event) { defer func() { if r := recover(); r != nil { if s.opts.OnHandlerError != nil { s.opts.OnHandlerError(reg.handlerName(), e.Code, errFromPanic(r)) } } }() if err := reg.Handler(ctx, *e, reg.Kwargs); err != nil { if s.opts.OnHandlerError != nil { s.opts.OnHandlerError(reg.handlerName(), e.Code, err) } }}A returned error travels as it was raised; the bus does not
re-code it, so whatever the handler minted is what gets logged. A
panic is projected to an error by errFromPanic: a panic value
that already is an error is adopted through
coded.FromExternal, and anything else becomes
X500016EVTHND: event handler
panicked {reason} {error}. That is the only code the bus itself mints
on this path.
Broadcast Failures
Section titled “Broadcast Failures”Cross-node broadcast failures are logged but not retried:
if err := bus.nodeBroadcaster.Broadcast(ctx, event); err != nil { slog.Debug("event broadcast failed", "error", err)}Configuration
Section titled “Configuration”The event bus has minimal configuration:
| Setting | Default | Description |
|---|---|---|
| Queue size | 10,000 | Maximum buffered events |
Monitoring
Section titled “Monitoring”Metrics
Section titled “Metrics”| Metric | Description |
|---|---|
shc_event_bus_events_total | Total events processed |
shc_event_bus_events_dropped | Events dropped (queue full) |
shc_event_bus_handlers_total | Handler registrations |
shc_event_bus_handler_errors | Handler execution errors |
Debugging
Section titled “Debugging”Enable debug logging to see event flow:
# Dev: run the daemon with debug loggingSHC_DEBUG=true make dev
# Production: edit the systemd unit to add the env var, then restartsudo systemctl edit shc # add: Environment=SHC_DEBUG=truesudo systemctl restart shcBest Practices
Section titled “Best Practices”-
Use appropriate dispatch modes
- SINGLE for exactly-once execution
- BROADCAST for local cache/state updates
- LEADER for cluster coordination
-
Keep handlers fast
- Long-running work should spawn goroutines, not block
- Return quickly from handlers
-
Don’t rely on event delivery
- Events are fire-and-forget
- Critical state should use polling/DB queries
-
Use operation_id for SSE progress
- operation_id equals trace_id for the request
- SSE clients subscribe to
GET /api/method/shc.operation.stream/{operation_id}
-
Handle errors gracefully
- Handler errors are logged, not fatal
- Check for nil when context might be missing
File Structure
Section titled “File Structure”modules/events/├── service.go # EventBus, Event, EventBusComponent├── broadcaster.go # NodeBroadcaster for cross-node events├── router.go # SSE stream handlers├── rpc.go # events.deliver, the cross-node receive function├── handlers.go # Handler registration, DispatchMode└── models.go # Event struct, NodeExecution