Metrics
Two places hold numbers about the cluster. The daemon holds the last few
seconds of every container’s CPU and memory, which shc status --resources
prints. The system OpenObserve on obs01 holds everything the collectors
shipped: host metrics, container metrics, container logs and the daemon’s own
log lines, kept for days. The daemon ships there on its own, the
managed default; otel.provider would name an otel.providers entry
instead.
$ shc config set otel.providers.remote.endpoint https://otlp.example.com:4318$ shc config set otel.provider remoteThree more rows shape what is shipped. otel.filter.quiet drops the
container log lines that match a pattern before they leave the node, and
otel.filter.rules takes a raw OTTL condition for what a pattern cannot say;
both are scoped like any config row: -t prod quiets one tenant’s
deployments and -s web -e prod one deployment, and a row set with no -t,
-s, -e or --scope is global.
otel.enabled false is the master switch: it takes the collector off every
node.
$ shc config set otel.filter.quiet 'GET /healthz' --scope tenant -t prod$ shc config set otel.filter.rules 'IsMatch(body, "^DEBUG")' --scope tenant -t prod$ shc config set otel.enabled false$ shc status --resourceslast polled 2026-10-09 07:03:04 UTCNODE STATE CPU% MEM% DISK% LOADhome01 healthy 31.8 37.5 17.9 2.63obs01 healthy 6.1 6.6 2.5 0.17
all 285 queues green on 2 nodes
TENANT STACK ENVIRONMENT STATE SERVICES NODES LABELS NO HEALTHCHECKmaster keycloak default healthy 1/1 home01 app=keycloak environment=default stack=keycloak tenant=master -master openobserve default healthy 1/1 obs01 app=openobserve environment=default stack=openobserve tenant=master 1master postgres default healthy 2/2 home01 app=postgres environment=default stack=postgres tenant=master -master traefik default healthy 15/15 home01, obs01 app=traefik environment=default stack=traefik tenant=master -prod cloud prod healthy 4/4 home01 app=nextcloud environment=prod stack=cloud tenant=prod 1prod db prod healthy 2/2 home01 app=postgres environment=prod stack=db tenant=prod -prod pgweb prod healthy 1/1 home01 app=/root/learn/pgweb environment=prod stack=pgweb tenant=prod -prod placed prod healthy 1/1 home01 app=notes environment=prod stack=placed tenant=prod tier=learn -prod store prod healthy 1/1 home01 app=/root/learn/versitygw environment=prod stack=store tenant=prod -prod web prod healthy 1/1 home01 app=/root/learn/nginx environment=prod stack=web tenant=prod 1
STACK ENVIRONMENT CPU MEMORY MEASUREDkeycloak default 0.43% 1.03 GiB 1/1openobserve default 3.08% 636.6 MiB 1/1postgres default 2.68% 164.0 MiB 2/2traefik default 7.49% 853.4 MiB 15/15cloud prod 0.81% 1.15 GiB 4/4db prod 0.65% 89.5 MiB 2/2pgweb prod 0.00% 8.5 MiB 1/1placed prod 0.64% 77.6 MiB 1/1store prod 0.05% 85.1 MiB 1/1web prod 0.00% 3.0 MiB 1/1
Published portsmaster/default/traefik traefik 80/tcp listen=node=@(service=traefik) nodes: home01 local, obs01 localmaster/default/traefik traefik 443/tcp listen=node=@(service=traefik) nodes: home01 local, obs01 local...2 nodes, 10 deploymentsOpenObserve listens on port 5080 inside its deployment. Forward that port to
this machine and sign in with the admin user and the password page 00 set.
shc forward holds the tunnel open until it is stopped, so run it in tmux.
$ tmux new -d -s fwd shc forward -t master openobserve 5080=15080$ curl -s -u [email protected]:P@ssw0rd http://127.0.0.1:15080/healthz{"status":"ok"}A query is a SQL statement over a stream, sent as JSON. Metrics are one
stream per series; this one asks the newest memory reading of web’s
container. The time window is in microseconds since the epoch, wide enough
to cover the walk.
{ "query": { "sql": "SELECT name, value FROM \"container_memory_usage_bytes\" WHERE name LIKE '%prod_prod_web%' ORDER BY _timestamp DESC", "start_time": 1700000000000000, "end_time": 2000000000000000, "from": 0, "size": 1 }}$ curl -s -u [email protected]:P@ssw0rd --json @/root/learn/metrics-query.json 'http://127.0.0.1:15080/api/default/_search?type=metrics'{"took":11,"took_detail":{"total":11,"cache_took":0,"file_list_took":0,"wait_in_queue":0,"idx_took":0,"search_took":9},"hits":[{"_timestamp":1791561157022238,"name":"prod_prod_web-nginx-r0-nginx-r0-1","value":3096576.0}],"total":1,"from":0,"size":1,"cached_ratio":0,"scan_size":1,"idx_scan_size":0,"scan_records":1630,"trace_id":"01a1215e57c97520be8f4f2e441bfc7a","is_partial":false,"result_cache_ratio":0,"work_group":"short","order_by":"desc","order_by_metadata":[["_timestamp","desc"]],"is_histogram_eligible":true,"peak_memory_usage":0.0}Now make one change and read it three ways. Restart web; the daemon raises
an event, nginx writes its startup lines, and the container’s memory reading
starts again from a fresh process.
$ shc restart webrestarting deployment "web"✔ restarted deployment "web"$ shc events ls -s web -T docker -n 22026-10-10T18:47:39Z home01 container prod_prod_web-nginx-r0-nginx-r0-1 exited with code 0, deployment prod/web/prod2026-10-10T18:47:40Z home01 container prod_prod_web-nginx-r0-nginx-r0-1 started, deployment prod/web/prod$ shc logs -s web -n 2nginx-r0 | 2026/10/10 18:47:40 [notice] 1#1: start worker process 23nginx-r0 | 2026/10/10 18:47:40 [notice] 1#1: start worker process 24Container log lines land in the default logs stream, each stamped with
the tenant, deployment and environment it came from. Ask for the newest
line about web: it is the last startup line shc logs printed.
{ "query": { "sql": "SELECT shc_tenant, shc_stack, body FROM \"default\" WHERE shc_stack = 'web' ORDER BY _timestamp DESC", "start_time": 1700000000000000, "end_time": 2000000000000000, "from": 0, "size": 1 }}$ curl -s -u [email protected]:P@ssw0rd --json @/root/learn/logs-query.json 'http://127.0.0.1:15080/api/default/_search?type=logs'{"took":22,"took_detail":{"total":22,"cache_took":0,"file_list_took":0,"wait_in_queue":0,"idx_took":0,"search_took":20},"hits":[{"_timestamp":1791561198675937,"shc_tenant":"prod","shc_stack":"web","body":"2026/10/09 15:53:18 [notice] 1#1: start worker process 23"}],"total":1,"from":0,"size":1,"cached_ratio":0,"scan_size":17,"idx_scan_size":0,"scan_records":22634,"trace_id":"01a1215e90827202aec8123a44243c8a","is_partial":false,"result_cache_ratio":0,"work_group":"short","order_by":"desc","order_by_metadata":[["_timestamp","desc"]],"is_histogram_eligible":true,"peak_memory_usage":2.0}shc support bundle collects all of this into one archive for someone else
to read: a dump of the telemetry store for the window --since names, capped
at --max-records, the state snapshot, every in-scope deployment’s render
with secrets left as references, the certificate inventory and the fabric
state. Nothing is uploaded; --file is where it lands. --quick skips the
telemetry dump.
$ shc support bundle --since 1h --max-records 10000 --file /root/learn/bundle.tar.gz✔ wrote support bundle "/root/learn/bundle.tar.gz", 56518 bytes$ shc support bundle --quick --file /root/learn/bundle-quick.tar.gz✔ wrote support bundle "/root/learn/bundle-quick.tar.gz", 56399 bytes$ tar tzf /root/learn/bundle-quick.tar.gzstate/status.jsonstate/license.jsonstate/deployments.jsonrender/master/keycloak/default/compose.yamlrender/master/keycloak/default/manifest.jsonrender/master/openobserve/default/compose.yamlrender/master/openobserve/default/manifest.jsonrender/master/postgres/default/compose.yamlrender/master/postgres/default/manifest.jsonrender/master/traefik/default/compose.yamlrender/master/traefik/default/manifest.jsonrender/prod/cloud/prod/compose.yamlrender/prod/cloud/prod/manifest.jsonrender/prod/db/prod/compose.yamlrender/prod/db/prod/manifest.jsonrender/prod/pgweb/prod/compose.yamlrender/prod/pgweb/prod/manifest.jsonrender/prod/placed/prod/compose.yamlrender/prod/placed/prod/manifest.jsonrender/prod/store/prod/compose.yamlrender/prod/store/prod/manifest.jsonrender/prod/web/prod/compose.yamlrender/prod/web/prod/manifest.jsoncerts/inventory.jsonfabric/wireguard.jsonmanifest.jsonA container’s readings reach OpenObserve once a minute, so the reading
taken right after the restart could still be the old process’s. Two
minutes on, ask for web’s newest memory reading again.
$ curl -s -u [email protected]:P@ssw0rd --json @/root/learn/metrics-query.json 'http://127.0.0.1:15080/api/default/_search?type=metrics'{"took":10,"took_detail":{"total":10,"cache_took":0,"file_list_took":0,"wait_in_queue":0,"idx_took":0,"search_took":9},"hits":[{"_timestamp":1791561277018281,"name":"prod_prod_web-nginx-r0-nginx-r0-1","value":3072000.0}],"total":1,"from":0,"size":1,"cached_ratio":0,"scan_size":1,"idx_scan_size":0,"scan_records":1767,"trace_id":"01a1216085cf7ac180c7dd6612f0700b","is_partial":false,"result_cache_ratio":0,"work_group":"short","order_by":"desc","order_by_metadata":[["_timestamp","desc"]],"is_histogram_eligible":true,"peak_memory_usage":0.0}The same restart is two docker events in shc events, two lines in shc logs,
its startup line in the logs stream and a memory reading of the new process
in OpenObserve.
An AI client reads the same cluster through shc mcp serve, which offers
the daemon’s API as MCP tools on its standard input and output. The
daemon’s own permissions apply to every call, and --read-only offers the
reading tools alone: search, status, logs and the lists.
$ shc mcp serve --read-onlyThe admin password is the last thing to read here, because changing it is
not a config change. OpenObserve reads modules.openobserve.vars.password
once, when its store is made, so the row page 00 set no longer reaches the
running store. shc system openobserve reset-admin-password does: it
changes the password in the running store, then the vault record the
collectors sign in with, then updates the deployment. --password names the
new one; without it SHC makes one and prints it once. Typed on home01, the
change is handed to obs01, which holds the store.
$ shc system openobserve reset-admin-password --password N3w-P@ssw0rd✔ changed the system openobserve admin passwordThe update replaced OpenObserve’s container, and the tunnel went to the old one. Open it again and ask for the list of log streams twice: the old password is refused and the new one signs in.
$ tmux kill-session -t fwd$ tmux new -d -s fwd shc forward -t master openobserve 5080=15080$ curl -s -o /dev/null -w '%{http_code}' -u [email protected]:N3w-P@ssw0rd 'http://127.0.0.1:15080/api/default/streams?type=logs'200$ curl -s -o /dev/null -w '%{http_code}' -u [email protected]:P@ssw0rd 'http://127.0.0.1:15080/api/default/streams?type=logs'401Put the example password back, so the pages after this one find the lab as page 00 left it, and close the tunnel.
$ shc system openobserve reset-admin-password --password P@ssw0rd✔ changed the system openobserve admin password$ tmux kill-session -t fwdThis page leaves two query files and two archives under ~/learn on home01
and nothing else changed. The HUD tutorial starts from here.
Next: The HUD.