Skip to content
SHC Docs

SHC test suite

Test suite for Self-Hosted Cloud (SHC).

tests/
├── unit/ # Go unit tests (plus in-package tests beside the code)
├── integration/ # Go integration tests (build tag `integration`)
├── parity/ # Golden-scenario CLI-surface pinning
├── e2e/ # End-to-end tests (single-node, BATS)
├── proxmox/ # Multi-node E2E on Proxmox VMs (BATS + terraform)
├── hetzner/ # Live hcloud E2E lane (costs money)
├── clouds/ # Per-cloud live smoke lanes (aws, azure, …; cost money)
├── tui/ # HUD tests (tui-test, pnpm)
├── manual/ # Hand-run scripts
└── TESTBOOK.md # Manual QA runbook (mixed-runtime 3-node cluster)
Terminal window
# Go tests (unit + integration)
make test
# Single-node E2E suite (builds bin/shc + bin/shcd, boots a dev daemon)
make test/e2e
# Multi-node scenarios on a standing rig cell
make -C tests/rig run CELL=cell1

Single-node end-to-end tests using BATS, driven by tests/e2e/scripts/test.sh (the make test/e2e entrypoint). The harness launches the freshly-built bin/shcd + bin/shc directly, no installation required.

Test FilePurpose
test_workflows.batsMain happy-path workflow: repo setup → install → operations (info, backup, clone, …) → uninstall
test_api_audit.batsAPI endpoint availability + audit log viewing/filtering/output formats
test_config_vault.batsConfig, vault, and node smoke tests
test_container_state.batsContainer state synchronization
test_dns.batsManaged DNS against a real PowerDNS container (connections-first path)
test_internal_api_bind.batsPublic API loopback bind in the e2e harness
test_cluster_ca_rotation.batsCluster-CA rotation (single node)
test_cluster_ca_rotation_global.batsGlobal-root CA rotation (single node)

What’s NOT covered here (use the rig): multi-node operations, cross-node move/clone, shared storage, quotas on a live cluster.

Note: apps that need real database containers can fail in the local sandbox; live backup/restore coverage belongs to the rig.

The standing two-site rig (SHC-1123): cells of real nodes, each cell described by tests/rig/cells/<cell>.env, and the scenarios under tests/rig/scenarios/ (a far node joining, moves with consumers, drain, a WAN cut, a control-plane node or a whole location lost). s11 to s17 cover what the retired Proxmox tier parked (SHC-1710): exec, logs and forward across nodes, clone and moves, drain against pins, CephFS on followers, a LAN leaf served from another node, a node rejoining, and one node’s deployment stopped while another serves. They publish their own fixtures into the cell’s rig repo through tests/rig/apps.sh. s14 needs CEPH_DEVICE and cephadm on the home nodes, and is INCONCLUSIVE without them. s18 is the restore drill (SHC-759): the catalogue postgres, seeded, backed up, changed and restored by id, passing only when the seed reads back and the audit chain holds A200402BKPRST for that backup.

Terminal window
make -C tests/rig up CELL=cell1 DEB=dist/shc_<version>_amd64.deb # roll a cell onto a deb
make -C tests/rig run CELL=cell1 # run the scenarios
make -C tests/rig run CELL=cell1 DISRUPTIVE=1 S="s08 s09" # the disruptive ones
make -C tests/rig health CELL=cell1 # is the cell healthy
Terminal window
make test/unit # hermetic default lane (CGO_ENABLED=0);
# covers ./cmd, ./internal, ./migrations, ./tests/unit
make test/integration # build tag `integration`, race detector on
make test/docsgen # build tag `docsgen`: scripts/docsgen, the docs generators
make test/portaldev # build tag shc_portal_dev: tests/unit/modules/portal and modules/license, the dev portal root and the dev broker key
make test # all of them, with the bats suites and test/ce

Every Go lane runs with -timeout $(GO_TEST_TIMEOUT), 40 minutes by default. Go’s own default is 10 minutes per package, and tests/unit/modules/app has taken 13 to 22 minutes on a loaded machine: the default killed it before it printed a result, which hid 36 red tests (SHC-728). Running a slow package by hand, pass the same flag: go test -timeout 40m ./tests/unit/modules/app/.

Live-engine unit tests (rqlited process, cluster join, engine flip) skip unless a rqlited binary is resolvable; point SHC_RQLITED_BIN at one (or make vendor/rqlite on linux) to include them.

Golden CLI-surface scenarios (tests/parity/)

Section titled “Golden CLI-surface scenarios (tests/parity/)”
Terminal window
make parity-golden # golden CLI-surface scenarios against bin/shc

(The manifest half of the old Go-vs-Python parity firewall (parity-check + migration/manifest.yaml) was retired once the port was complete; the golden lane stays because it pins the CLI surface against regression.)

make test/e2e is the entrypoint; the script accepts a file or filter:

Terminal window
./tests/e2e/scripts/test.sh # all tests
./tests/e2e/scripts/test.sh test_workflows.bats # one file
./tests/e2e/scripts/test.sh --filter dns # files matching a pattern
./tests/e2e/scripts/test.sh --no-cleanup # keep the server after tests
make test/e2e/no-cleanup # same, via make
make test/e2e/clean # tear residual state down

Filtering inside a file uses standard bats flags:

Terminal window
bats tests/e2e/test_workflows.bats --filter "backup"
bats tests/e2e/test_workflows.bats --tap

(Running bats directly assumes make build has produced current bin/shc + bin/shcd; prefer the script, which handles server lifecycle and cleanup.)

The GitLab pipeline (.gitlab-ci.yml) builds and publishes: packages, container images, terraform modules/provider, the apps index, and pages. It runs no verification suites. The two gates it does keep are build work rather than checks: the packaging checks on the binaries and debs it is about to publish, and the leak scan over the publishable corpus.

Everything else runs locally:

Terminal window
make ci/checks # every gate the pipeline no longer runs
make lint # formatting, linters, architecture fitness
make test # unit + integration

E2E lanes are run from a workstation (make test/e2e) or against infrastructure (make -C tests/rig run CELL=<cell>, make test/hetzner, make test/<cloud>; the latter two cost money).

VariablePurposeDefault
SHC_VAULT_PASSWORDVault password the harness seeds/unlocks withshc
NO_CLEANUP1 = keep server + state after make test/e2e0
SHC_RQLITED_BINrqlited binary for live-engine unit testsnone

See environment-variables.md for the full SHC_* surface (including the test-only overrides).

#!/usr/bin/env bats
load "scripts/helpers.sh"
setup_file() {
# One-time setup (helpers.sh's setup() runs per-test)
}
@test "TEST-001: description of test" {
run shc ls
[ "$status" -eq 0 ]
assert_output_contains "expected"
}

tests/e2e/scripts/helpers.sh provides the shared harness, a sampling:

Terminal window
start_server / stop_server # dev daemon lifecycle
shc / shc_nontty / api_call # CLI + API wrappers
assert_output_contains "text"
assert_output_not_contains "text"
validate_json
wait_for_stack / wait_for_stack_status / wait_for_stack_gone
wait_for_container / wait_for_repo
cleanup_stacks / cleanup_docker_containers
Terminal window
# The harness logs the dev daemon here (STATE_DIR from helpers.sh):
tail -f /tmp/shc-e2e/server.log
# Check status of a harness-launched daemon
./tests/e2e/scripts/status.sh
Terminal window
# Verbose output
bats tests/e2e/test_workflows.bats -t
# Keep state for post-mortem
make test/e2e/no-cleanup
./tests/e2e/scripts/logs.sh
Terminal window
# Is every node of the cell healthy
make -C tests/rig health CELL=cell1
  • tests/TESTBOOK.md: mixed-runtime 3-node cluster QA runbook (postgres/keycloak/paperless-ngx, nip.io hostnames).
  • tests/manual/: hand-run scripts for scenarios the suites don’t automate.