SHC test suite
Test suite for Self-Hosted Cloud (SHC).
Test Organization
Section titled “Test Organization”tests/├── unit/ # Go unit tests (plus in-package tests beside the code)├── integration/ # Go integration tests (build tag `integration`)├── parity/ # Golden-scenario CLI-surface pinning├── e2e/ # End-to-end tests (single-node, BATS)├── proxmox/ # Multi-node E2E on Proxmox VMs (BATS + terraform)├── hetzner/ # Live hcloud E2E lane (costs money)├── clouds/ # Per-cloud live smoke lanes (aws, azure, …; cost money)├── tui/ # HUD tests (tui-test, pnpm)├── manual/ # Hand-run scripts└── TESTBOOK.md # Manual QA runbook (mixed-runtime 3-node cluster)Quick Start
Section titled “Quick Start”# Go tests (unit + integration)make test
# Single-node E2E suite (builds bin/shc + bin/shcd, boots a dev daemon)make test/e2e
# Multi-node scenarios on a standing rig cellmake -C tests/rig run CELL=cell1Test Categories
Section titled “Test Categories”E2E Tests (tests/e2e/)
Section titled “E2E Tests (tests/e2e/)”Single-node end-to-end tests using BATS, driven by
tests/e2e/scripts/test.sh (the make test/e2e entrypoint). The
harness launches the freshly-built bin/shcd + bin/shc directly,
no installation required.
| Test File | Purpose |
|---|---|
test_workflows.bats | Main happy-path workflow: repo setup → install → operations (info, backup, clone, …) → uninstall |
test_api_audit.bats | API endpoint availability + audit log viewing/filtering/output formats |
test_config_vault.bats | Config, vault, and node smoke tests |
test_container_state.bats | Container state synchronization |
test_dns.bats | Managed DNS against a real PowerDNS container (connections-first path) |
test_internal_api_bind.bats | Public API loopback bind in the e2e harness |
test_cluster_ca_rotation.bats | Cluster-CA rotation (single node) |
test_cluster_ca_rotation_global.bats | Global-root CA rotation (single node) |
What’s NOT covered here (use the rig): multi-node operations, cross-node move/clone, shared storage, quotas on a live cluster.
Note: apps that need real database containers can fail in the local sandbox; live backup/restore coverage belongs to the rig.
Rig scenarios (tests/rig/)
Section titled “Rig scenarios (tests/rig/)”The standing two-site rig (SHC-1123): cells of real nodes, each cell
described by tests/rig/cells/<cell>.env, and the scenarios under
tests/rig/scenarios/ (a far node joining, moves with consumers, drain,
a WAN cut, a control-plane node or a whole location lost). s11 to s17 cover
what the retired Proxmox tier parked (SHC-1710): exec, logs and forward
across nodes, clone and moves, drain against pins, CephFS on followers, a
LAN leaf served from another node, a node rejoining, and one node’s
deployment stopped while another serves. They publish their own fixtures
into the cell’s rig repo through tests/rig/apps.sh. s14 needs
CEPH_DEVICE and cephadm on the home nodes, and is INCONCLUSIVE without
them. s18 is the restore drill (SHC-759): the catalogue postgres, seeded,
backed up, changed and restored by id, passing only when the seed reads back
and the audit chain holds A200402BKPRST for that backup.
make -C tests/rig up CELL=cell1 DEB=dist/shc_<version>_amd64.deb # roll a cell onto a debmake -C tests/rig run CELL=cell1 # run the scenariosmake -C tests/rig run CELL=cell1 DISRUPTIVE=1 S="s08 s09" # the disruptive onesmake -C tests/rig health CELL=cell1 # is the cell healthyGo Tests
Section titled “Go Tests”make test/unit # hermetic default lane (CGO_ENABLED=0); # covers ./cmd, ./internal, ./migrations, ./tests/unitmake test/integration # build tag `integration`, race detector onmake test/docsgen # build tag `docsgen`: scripts/docsgen, the docs generatorsmake test/portaldev # build tag shc_portal_dev: tests/unit/modules/portal and modules/license, the dev portal root and the dev broker keymake test # all of them, with the bats suites and test/ceEvery Go lane runs with -timeout $(GO_TEST_TIMEOUT), 40 minutes by
default. Go’s own default is 10 minutes per package, and
tests/unit/modules/app has taken 13 to 22 minutes on a loaded machine:
the default killed it before it printed a result, which hid 36 red tests
(SHC-728). Running a slow package by hand, pass the same flag:
go test -timeout 40m ./tests/unit/modules/app/.
Live-engine unit tests (rqlited process, cluster join, engine flip)
skip unless a rqlited binary is resolvable; point SHC_RQLITED_BIN
at one (or make vendor/rqlite on linux) to include them.
Golden CLI-surface scenarios (tests/parity/)
Section titled “Golden CLI-surface scenarios (tests/parity/)”make parity-golden # golden CLI-surface scenarios against bin/shc(The manifest half of the old Go-vs-Python parity firewall
(parity-check + migration/manifest.yaml) was retired once the port
was complete; the golden lane stays because it pins the CLI surface
against regression.)
Running E2E Tests
Section titled “Running E2E Tests”make test/e2e is the entrypoint; the script accepts a file or filter:
./tests/e2e/scripts/test.sh # all tests./tests/e2e/scripts/test.sh test_workflows.bats # one file./tests/e2e/scripts/test.sh --filter dns # files matching a pattern./tests/e2e/scripts/test.sh --no-cleanup # keep the server after testsmake test/e2e/no-cleanup # same, via makemake test/e2e/clean # tear residual state downFiltering inside a file uses standard bats flags:
bats tests/e2e/test_workflows.bats --filter "backup"bats tests/e2e/test_workflows.bats --tap(Running bats directly assumes make build has produced current
bin/shc + bin/shcd; prefer the script, which handles server
lifecycle and cleanup.)
CI Integration
Section titled “CI Integration”The GitLab pipeline (.gitlab-ci.yml) builds and publishes: packages,
container images, terraform modules/provider, the apps index, and pages.
It runs no verification suites. The two gates it does keep are build
work rather than checks: the packaging checks on the binaries and
debs it is about to publish, and the leak scan over the publishable
corpus.
Everything else runs locally:
make ci/checks # every gate the pipeline no longer runsmake lint # formatting, linters, architecture fitnessmake test # unit + integrationE2E lanes are run from a workstation (make test/e2e) or against
infrastructure (make -C tests/rig run CELL=<cell>, make test/hetzner,
make test/<cloud>; the latter two cost money).
Test Environment Variables
Section titled “Test Environment Variables”| Variable | Purpose | Default |
|---|---|---|
SHC_VAULT_PASSWORD | Vault password the harness seeds/unlocks with | shc |
NO_CLEANUP | 1 = keep server + state after make test/e2e | 0 |
SHC_RQLITED_BIN | rqlited binary for live-engine unit tests | none |
See environment-variables.md
for the full SHC_* surface (including the test-only overrides).
Writing New Tests
Section titled “Writing New Tests”BATS Test Template
Section titled “BATS Test Template”#!/usr/bin/env bats
load "scripts/helpers.sh"
setup_file() { # One-time setup (helpers.sh's setup() runs per-test)}
@test "TEST-001: description of test" { run shc ls [ "$status" -eq 0 ] assert_output_contains "expected"}Helper Functions
Section titled “Helper Functions”tests/e2e/scripts/helpers.sh provides the shared harness, a
sampling:
start_server / stop_server # dev daemon lifecycleshc / shc_nontty / api_call # CLI + API wrappersassert_output_contains "text"assert_output_not_contains "text"validate_jsonwait_for_stack / wait_for_stack_status / wait_for_stack_gonewait_for_container / wait_for_repocleanup_stacks / cleanup_docker_containersTroubleshooting
Section titled “Troubleshooting”Server Not Starting
Section titled “Server Not Starting”# The harness logs the dev daemon here (STATE_DIR from helpers.sh):tail -f /tmp/shc-e2e/server.log
# Check status of a harness-launched daemon./tests/e2e/scripts/status.shTests Failing
Section titled “Tests Failing”# Verbose outputbats tests/e2e/test_workflows.bats -t
# Keep state for post-mortemmake test/e2e/no-cleanup./tests/e2e/scripts/logs.shRig Scenarios Failing
Section titled “Rig Scenarios Failing”# Is every node of the cell healthymake -C tests/rig health CELL=cell1Manual Testing
Section titled “Manual Testing”tests/TESTBOOK.md: mixed-runtime 3-node cluster QA runbook (postgres/keycloak/paperless-ngx, nip.io hostnames).tests/manual/: hand-run scripts for scenarios the suites don’t automate.