Skip to content
SHC Docs

Portal secret custody: operating a bench that holds customer cloud credentials

Applies to: the Portal bench (my.selfhostedcloud.org) once the terraform runner is live. Design of record: docs/proposals/2026-08-15-portal-terraform-runner.md.

The owner ruled on 2026-08-15 that customer cloud credentials live in Frappe Password fields rather than OpenBao. That ruling is sound: fieldtype Password lands in the __Auth table, Fernet-encrypted (AES-128-CBC + HMAC) on the site’s encryption_key, readable only through get_password(), invisible to frappe.db.get_value, to list views, to the REST document payload and to the desk UI.

It also changes three things about how this bench must be operated. The runner implements what code can implement; this document is the rest, and it is not optional; proposal §4b records these as chosen properties, which only holds if somebody actually operates them.


encryption_key is site-wide. It decrypts every customer’s cloud credentials and every cluster’s harvested SSH key, and by default it sits in site_config.json on the same host as the database. A single host compromise, or a leaked backup pair (DB dump + site config), exposes the whole book of business at once, not one customer.

Operate it this way:

  • Hold encryption_key in the container’s environment / secret mount (FRAPPE_ENCRYPTION_KEY), not in the on-disk site_config.json, wherever the platform allows it.

  • Treat database + site config as ONE custody boundary. They are never backed up to the same destination, never handed to the same operator for a restore, and never both present on a laptop.

  • The runner stamps encryption_key_custody (env / site_config / unknown) on every Terraform Run row. That field is the estate’s actual posture, queryable, per run. If it says site_config in production, that is a finding, not a default.

    select encryption_key_custody, count(*)
    from `tabTerraform Run`
    where creation > date_sub(now(), interval 30 day)
    group by 1;

Every routine backup, every replica, every “let me copy prod into staging to reproduce it” carries __Auth. That tarball is not a database backup any more; it is a credential archive.

Operate it this way:

  • Portal bench backups are encrypted at rest and access-controlled like the secrets they contain, not like a database backup.

  • No prod → staging/dev data copy without scrubbing __Auth. The check is frappe/apps/marketplace/scripts/portal-backup-guard.sh:

    Terminal window
    # refuse a dump that still carries __Auth into a non-prod site
    sh scripts/portal-backup-guard.sh dump.sql.gz staging
    # produce a restorable dump with the __Auth ROWS removed (schema kept)
    sh scripts/portal-backup-guard.sh --scrub dump.sql.gz > scrubbed.sql

    A staging portal that can reach a customer’s cloud account is a staging portal that can bill a customer’s cloud account.

  • Restore drills use the scrubbed dump. If a drill needs real secrets to pass, the drill is testing the wrong thing.

3. There is no read audit, no lease, no revocation

Section titled “3. There is no read audit, no lease, no revocation”

get_password() leaves no trail. The framework cannot tell you who read which secret when, and it cannot revoke one.

The mitigation is the run ledger, and it works at the job level:

  • every Terraform Run row records secrets_read: the credential field names the run read, never values;

  • so “which secrets did that job touch?” is answerable, and “which jobs touched this customer’s credentials?” is a query:

    select name, action, status, started_at, secrets_read
    from `tabTerraform Run`
    where customer = %(customer)s
    order by started_at desc;
  • revocation is manual and it is the customer’s cloud console, not ours. When a credential is suspected exposed, rotate it at the cloud provider and re-enter it in the portal. Portal Cloud Credential.write_only defaults on, so the portal never renders it back; a rotation costs a re-entry, which is the trade §5.2 of the proposal names.


What the code already enforces (do not re-check by hand)

Section titled “What the code already enforces (do not re-check by hand)”
PropertyWhereGuard test
Credentials only in the process environmentprovisioning/runenv.pytest_secret_injection.py
Never a tfvars file, never argvprovisioning/runenv.py, runner.pytest_secret_injection.py
TF_LOG never DEBUG/TRACE/JSON outside a diagnostic runprovisioning/runenv.pytest_log_policy.py
Injected secrets scrubbed from the log in raw/base64/JSON formprovisioning/redact.pytest_redaction_canary.py
Per-run workdir on tmpfs, destroyed on success/failure/killprovisioning/workdir.pytest_workdir.py
Failed apply reaps; failed reap opens a leak recordprovisioning/runner.pytest_reap_and_tags.py
Every resource tagged in house KEY form; selector derived from itprovisioning/tags.pytest_reap_and_tags.py
Provider + module pinned to an exact releaseprovisioning/versions.pytest_ledger_and_versions.py

CI runs the Go-driven scenarios in tests/unit/portal/clouds_test.go and provisioning_test.go. The Python drivers use fake HTTP and subprocesses; the state driver exercises real AES-256-GCM. The older marketplace Python unittests remain available locally and are not reintroduced into CI.

4. State has its own bucket (RULED 2026-11-01)

Section titled “4. State has its own bucket (RULED 2026-11-01)”

Terraform state holds the SSH key and vault password together. It is stored in a dedicated bucket in the existing bitspur B2 account, through its S3-compatible API. Each cluster has a random 32-byte data key in the Terraform State.data_key Password field. State is encrypted with AES-256-GCM using the cluster name as AAD; the object key is <prefix>/<cluster>.tfstate.enc. The row also keeps serial, byte count and lock metadata.

The DB backup carries keys without state. The bucket carries ciphertext without keys. Keep the bucket and bench backup access separate. The existing credential and harvested-secret backup precautions above still apply. There is no __Auth state fallback and no chunked state DocType.

Configure portal_state_bucket, portal_state_endpoint, portal_state_region and portal_state_prefix in site config. Supply PORTAL_STATE_KEY_ID and PORTAL_STATE_APP_KEY through vault-backed worker environment variables. Clay supplies their values; deployment does not mint them. A run with missing bucket configuration fails before a lock, workdir or Terraform command and names the missing keys in its redacted log.

The marketplace stack’s provisioning variables write the non-secret site configuration during bench-configure. Provisioning is closed by default: portal_provisioning_open must be enabled, or the caller must be a System Manager or administer an org in portal_provisioning_orgs. Connecting a cloud remains open to org admins. Set portal_cluster_domain, portal_provision_shc_version and portal_cluster_portal_url before a provision. The domain suffix is still an open-design placeholder.

OAuth client IDs live in portal_oauth_clients; the three PORTAL_OAUTH_<CLOUD>_CLIENT_SECRET values come from the deployment’s vault rows portal_oauth_<cloud>_client_secret, mounted on the web and worker containers as PORTAL_OAUTH_<CLOUD>_CLIENT_SECRET_FILE. Leave the provisioning.<cloud>ClientId variables empty until the clients exist. PORTAL_PROVISION_NETBIRD_SETUP_KEY is optional and seeded only in dev. Its presence opts into NetBird enrol; the environment name is not a gate. Registration always uses shc portal register - over SSH after apply, with the code carried on stdin rather than in any argv.

vault_password uses render.default_vault_password_mode = "generated", the 2026-11-01 class-B ruling. "reference" remains implemented as an alternative. The run ledger records the chosen mode, preflight, refresh and enrol outcome. An enrol failure preserves the created cluster for diagnosis.

The current CLI has a registration prerequisite outside this backend: modules/portal/commands.go calls an in-process agent, while daemon/wiring/portal_wire.go publishes that agent only in the daemon. cmd/shc/main.go does not wire it. Until that CLI path is connected, a real post-apply registration can refuse with X503145PTLNYI. The runner records the enrol failure and keeps the cluster. The offline SSH fake verifies the invocation and secret handling, not successful live registration.