Skip to content
SHC Docs

Vault & secrets management

SHC’s vault is the single place where every secret used by a stack lives: database passwords, API keys, OIDC client secrets, webhook tokens. Secrets are referenced from app.yaml, vars.yaml, and compose.yaml using a ref+vault:// URI; SHC resolves and injects the real value at render time.

SHC supports 11 backend providers for the vault, so you can keep the default provider (encrypted in SHC’s state database), or delegate to an external secret manager (AWS Secrets Manager, Azure Key Vault, HashiCorp Vault, GCP, Bitwarden, Doppler, Infisical, 1Password, pass, SOPS).

Source of truth: modules/vault/service.go, modules/vault/crypto.go, modules/vault/providers/, core/config/secrets.go (ref URI parser).

For the full authorization / scoping model see architecture/security.md.


flowchart LR
compose["compose.yaml<br>PASSWORD:<br>ref+vault://.."] --> render["render pipeline<br>resolve_ref_async"]
render --> vault["vault<br>provider.get"]
vault --> injected["real plaintext<br>injected into compose env"]
  • Secrets are never stored in plaintext in app.yaml, vars.yaml, deployment files on disk, or compose before render.
  • Only the reference URI is persisted; the actual value lives in the vault provider.
  • References carry a scope (per-stack, per-environment, per-tenant, or global) so the same URI in different stacks resolves to different values.

Every SHC install gets a local vault provider out of the box: values are envelope-encrypted at rest in SHC’s state database. One Argon2id key-encryption key is derived from the master password at each unlock and held in memory; every secret carries its own random AES-256-GCM data key, wrapped by it. Stored values are tagged v2:.

There is exactly one at-rest format. The PBKDF2/Fernet format used before v0.0.16 has no reader; see the vault module for the retirement and the one-way make migrate step it requires.

Tenant isolation is database-level. Each tenant’s secrets are stored under its tenant row; reads are guarded by the same scope-matching rules the permission decorator uses; see security.md.

There is no separate vault-init step. The master password is set when the node is initialized, and only a verifier hash (PBKDF2) is stored in the state database; the password itself is never persisted.

Terminal window
shc init --vault-password-file /root/vault.pw --yes # non-interactive
shc init # prompts for it

--vault-password-file expects mode 0600 and takes precedence over --vault-password and SHC_VAULT_PASSWORD; pass - to read stdin.

SHC refuses writes to the vault while locked. On every daemon start shcd opens the sealed auto-unlock credential once and unlocks with it when the file is present. A missing file is the follower posture: the vault stays locked until an operator runs shc vault unlock (or, on a multi-node cluster, until a peer pushes the unlock).

shcd never reads SHC_VAULT_PASSWORD. That variable is the retired delivery path. If it is set in shcd’s environment, startup refuses with X400975VLTRFE and does not fall back to the variable or to the file. SHC_VAULT_PASSWORD remains a CLI input for shc init and shc vault only; those are a different process from shcd.

Source: subsystems/impl/vault.go.

The cold-start node keeps the master in one file, sealed with systemd-creds under the credential name vault-unlock:

Terminal window
# /etc/systemd/system/shc.service.d/vault-autounlock.cred (chmod 0600, root:root)
<systemd-creds output; no plaintext>

It is sealed to the node’s credential host key (/var/lib/systemd/credential.secret), and to its TPM2 as well when the node has a usable one. It is bound to no TPM PCR: a PCR 7 binding breaks on a Secure Boot or firmware update and would strand the node that breaks the cluster’s cold-start deadlock. No unit line names the file. shcd opens it itself at boot with systemd-creds decrypt, so the plaintext exists only in shcd’s memory. shc.service runs systemd-creds setup before shcd starts, which creates the host key once.

A sealed credential that will not open (the host key or the TPM it was sealed to changed, or the file is damaged) does not stop shcd. The daemon comes up with the vault locked and logs a warning naming the file. Unlock it with shc vault unlock, which seals a fresh credential, and the next restart auto-unlocks again. The vault.autounlock_at_rest row of shc doctor run -c vault reports a credential that no longer opens before a restart finds out.

Three genesis paths plant it, and they plant the same file: the terraform provider (writeVaultAutoUnlockDropIn), shc init when it is given the master password (--vault-password-file, --vault-password, or stdin), and the daemon when cold-start moves onto a node (plantLocalColdStart). The master reaches systemd-creds on stdin, never in argv, and every seal is opened again and compared before the file is written. On launchd the same paths write a System keychain item instead (see below). A resumed shc init (the node is already bootstrapped) re-plants it only when it unlocks the vault itself and the vault accepts the password. When the vault was already unlocked, or the unlock is refused, it plants nothing and, if the credential is missing, prints the repair: shc vault coldstart replant --node <node>. shc vault rotate and shc vault unlock keep an existing credential in step and do not create one.

On launchd the credential is a generic-password item in the System keychain (/Library/Keychains/System.keychain, service io.selfhostedcloud.shc.vault-unlock, account shcd). No other account can open it without an administrator’s authorization. The root launchd shcd reads it at boot. shc init and the cold-start replant write it through /usr/bin/security on stdin, so the password never appears in a process list, and read it back before it counts. Rotation and every unlock keep an existing item in step and never create one on a follower.

A Mac that has a /etc/shc/vault-unlock file (the plaintext form before the keychain) still unlocks from it once. The next unlock then moves the password into the keychain item and drops the file once the item reads back. vault.autounlock_at_rest flags the file until then.

systemd-creds arrived in systemd 250. Debian 12 (252), Debian 13 (257), Ubuntu 24.04 (255) and RHEL 9 (252) all seal. On an older systemd (Ubuntu 22.04 ships 249) SHC writes no credential at all, never a plaintext one: shc init and the provider warn (X200052VLTSKP), and that node’s vault needs shc vault unlock after every restart. vault.autounlock_at_rest keeps saying so.

v0.1.21 kept the master in plaintext beside a unit line that named it:

/etc/systemd/system/shc.service.d/vault-autounlock.conf
[Service]
LoadCredential=vault-unlock:/etc/systemd/system/shc.service.d/vault-autounlock.env

and vault-autounlock.env held the raw password and a newline. The upgrade converts it in place with no step from you. The first start of the new shcd still unlocks from that file. The repair that follows every unlock then seals the master and opens the seal back. It removes the .conf, reloads systemd, and removes the .env last. At every step, both the unit on disk and systemd’s cached copy name a file that is there. A v0.1.21 .conf naming a missing .env failed the unit with 243/CREDENTIALS. The sealed form names nothing, so that failure is gone. On a node that cannot seal, the v0.1.21 pair stays, and rotation keeps it in step.

Do not put the password in the unit environment. Environment=SHC_VAULT_PASSWORD=… and EnvironmentFile= both deliver it through the process environment. systemd publishes a unit’s resolved environment as a queryable D-Bus property, and /proc/<pid>/environ keeps that copy for the life of the process. A unit that still sets SHC_VAULT_PASSWORD makes shcd refuse to start (X400975VLTRFE). The package ships the guard at /usr/lib/shc/scripts/vault-password-not-queryable.sh; run it on any node to check.

On a multi-node cluster only the genesis node holds the credential: followers come up locked and pull the unlock from an already-unlocked peer.

The password is the credential’s bytes, minus one newline

Section titled “The password is the credential’s bytes, minus one newline”

The sealed plaintext is the password and one trailing newline. shcd trims exactly that newline and uses the rest as the password.

shc vault rotate still refuses a master that contains whitespace, a line break, a quote, a backslash, or a percent sign (X400965VLTDRI). Choose a master inside that set (random_password with special = false is what both live estates use).

Terminal window
shc vault # prompts for password, unlocks the vault
shc vault unlock # explicit alias of bare `shc vault`
shc vault --password "$PW" # unlock non-interactively
shc vault status # shows initialized + locked state

Source: modules/vault/commands/main.go.

The master password derives the KEK that wraps every per-value data key. Rotating it re-wraps every encrypted value (the payload ciphertext itself is untouched) and flips the replicated password verifier in the same atomic batch:

Terminal window
shc vault rotate # prompts for old + new

Do this after restoring from a backup from a different cluster, or when the old password may have leaked. Other provider types (AWS / HashiCorp / …) handle this in their own dashboards; SHC does not try to proxy it.

A rotation makes the unlock-key escrow receipt stale. Escrow the new master and record it again (next section).

Escrowing the unlock key and the backup key

Section titled “Escrowing the unlock key and the backup key”

Two keys decide whether an estate can be recovered after it is lost:

  • the unlock key, the vault master password. On the cold-start node it is in the sealed auto-unlock credential, which opens only on that node and is in no backup archive.
  • the backup key, the restic repository key backup.password names. shc init seals it into this vault and nowhere else. Every off-site archive needs it to open, and it does not change when the vault rotates.

Lose the estate with both inside it and every archive stays sealed. So copy each one out of band, somewhere outside the estate: a password manager, a printed sheet in a safe, a CI variable on another system. Keep the two apart if you can, so no single copy opens everything. SHC never holds, moves or checks that copy. You record that it exists:

Terminal window
shc vault escrow record --what unlock-key --where '1Password, vault Infra, item SHC master'
shc vault escrow record --what backup-key --where 'printed sheet, office safe' \
--note 'repository URL and bucket account on the same sheet'
shc vault escrow status # each key: ok, stale, missing or not needed
shc vault escrow verify # same table, exits non-zero unless every key is ok

A receipt holds the label, the note, the node and principal that recorded it, and the key’s generation. It never holds the key: a --where or --note that contains the master or the backup key is refused (X400985VLTESK), because receipts are replicated into every database backup.

The generation is what makes a receipt go stale:

ChangeStale receipt
shc vault rotateunlock-key
shc backup key rotate, or any write of the backup keybackup-key

Re-escrow the new key, then run record again; a second record for the same key replaces the first. The backup key needs no receipt when backup.password is unset or points at another secret provider, which holds it off the estate.

shc doctor runs vault.unlock_key_escrow on every node. It fails while a key that needs a receipt has none or a stale one, and its hint names the record command to run.


Wherever you’d write a plaintext secret, write a reference instead:

vars.yaml
adminPassword: "!ref+vault://admin_password"
# With explicit scope
apiKey: "ref+vault://shared_key?scope=tenant"
# Auto-generate on first install
sessionSecret: "ref+vault://session?autogen&length=32"
# Custom charset
dbPassword: "ref+vault://db?autogen&charset=alphanum&length=24"
ref+<provider>://<key>[?<params>]
params:
scope=<base|deployment|environment|local|stack|tenant>
autogen generate a value if missing
length=<n> autogen length (default 32)
charset=<alphanum|ascii> autogen charset (default printable)
<custom>=<value> forwarded to the provider

Parser: core/config/secrets.go (ParseSecretRef, SecretRef).

ScopeKey namespaceTypical use
deployment (default)(tenant, stack, environment)Per-install values: this is almost always what you want
environment(tenant, environment)Values shared across all stacks in one environment
stack(tenant, stack)Shared across environments of one stack
tenant(tenant,)Tenant-wide values (an API key for a single SaaS account)
localPer-nodeNode-local secrets (rare; see node-agent config)
baseGlobalCluster-wide defaults; use sparingly

A ref embedded in a larger value is resolved in place:

connection_string: "postgres://user:ref+vault://db_pass@host:5432/db"

The pattern matches ref+vault:// followed by non-whitespace characters (from secrets.go), so the ref ends at whitespace, quotes, or closing braces/brackets.

?autogen tells SHC: if this key has no value yet, generate one and store it. Every update, restart and redeploy of the deployment reuses the stored value, so auto-generated admin passwords stay stable across upgrades and restarts. An uninstall purges the deployment’s keys with its config. A later install under the same name is a new deployment and mints new ones, and shc recover -b with the uninstall’s safety backup brings the old ones back with the data.

Never commit ?autogen keys for user-facing credentials you need to hand back to the user; use shc vault get <scope> <key> to retrieve the generated value after install.

On install or update, an autogen vault ref in vars that supplies oidc.plug.yaml’s data.clientSecret gets a credential=deployment parameter. SHC persists the marked ref, not its resolved value. It reads deployment, stack, then environment scope. A row at any of those tiers wins; otherwise SHC creates a deployment-scoped row. Tenant and global rows are excluded, including when the ref previously requested a broader mint scope. Subsequent renders and read-only checks use the same restricted read.

Packages can also declare ?autogen&credential=deployment for another per-deployment credential. Ordinary refs and non-autogen OIDC refs retain their existing cascade. Existing deployment rows are reused.

If a deployment’s OIDC client secret came from a tenant or global row, whether in one environment or shared across several, the next install or update without a narrower row creates a separate secret for that deployment and supplies it to the OIDC socket. Coordinate consumer and provider reconciliation when rolling this change. To keep the old value, copy it to deployment scope first. To keep an explicit shared override, provision it at stack scope, or at environment scope for consumers in one environment. The tenant row is left untouched.


Terminal window
shc vault set <key> <value> [--scope stack] [--stack blog]
shc vault get <key> [--scope stack] [--stack blog]
shc vault ls [--scope tenant]
shc vault exists <key> [--stack blog]
shc vault rm <key> [--scope stack] [--stack blog]

These commands take --stack and --environment to select the scope tuple. set, get, ls and rm also take --scope. Source: modules/vault/commands/.

Terminal window
# Replace an autogenerated admin password
shc vault set admin_password "new-strong-password" \
--stack blog --environment prod
# Inspect everything at tenant scope for audit
shc vault ls --scope tenant
# Copy a secret from one environment to another
VAL=$(shc vault get api_key --stack myapp --environment staging)
shc vault set api_key "$VAL" --stack myapp --environment prod
# Rotate, then restart dependent stacks
shc vault set db_password "new-pw" --stack postgres --scope stack
shc restart blog && shc restart analytics && shc restart wiki

SHC can delegate vault reads/writes to an external system. Each provider is a type in modules/vault/providers/ with Get, Set, Delete, Exists, List methods.

ProviderURI prefixSource
Local (default, encrypted in the state database)ref+vault://providers/local.go
AWS Secrets Managerref+aws://providers/aws.go
Azure Key Vaultref+azure://providers/azure.go
GCP Secret Managerref+gcp://providers/gcp.go
HashiCorp Vaultref+hashicorp://providers/hashicorp.go
Bitwardenref+bitwarden://providers/bitwarden.go
Dopplerref+doppler://providers/doppler.go
Infisicalref+infisical://providers/infisical.go
1Password (Connect / CLI)ref+onepassword://providers/onepassword.go
pass (the standard Unix password manager)ref+pass://providers/pass.go
SOPSref+sops://providers/sops.go

You can use multiple providers simultaneously: the URI scheme picks which provider resolves a reference. Typical pattern:

  • local for app-generated secrets (autogen’d admin passwords, session keys): stays alive with the cluster, zero infra deps.
  • External provider for shared org credentials (GitHub tokens, SaaS API keys): values managed centrally, rotated outside SHC.

Each provider has its own config shape, usually credentials + an endpoint. Configure under vault.providers: in your user config. Example for HashiCorp Vault:

vault:
providers:
hashicorp:
enabled: true
address: "https://vault.example.com:8200"
token_file: "/etc/shc/vault-token"
path_prefix: "secret/shc/"

Then reference with the provider-specific prefix:

GITHUB_TOKEN: "ref+hashicorp://shared/github-token"

Exact config keys per provider live in the provider’s *Config struct, e.g. AWSConfig in providers/aws.go.


Local vault values are included in cluster-wide backups of the state database but protected by the same master password. Restoring a backup from a cluster with a different master password requires rotation before secrets become decryptable. See disaster-recovery.md for the restore procedure.

External-provider values are not SHC-backed; they live in the external system. SHC backs up only the references, and assumes the external system has its own DR story.

Every vault read/write emits an audit event (modules/vault/audit.go). See architecture/security.md#audit-trail and architecture/modules/audit.md.

The actual secret value is never in the audit event, only the key, scope, operation, and actor.

ErrorCauseFix
X401012VLTINVinvalid vault password: the supplied password does not open the vault (unlock, rotate)Reconfigure password_file / password_command, restart daemon
X500961VLTCRPvault crypto failure {op} {error}: the blob would not decrypt: wrong key, truncated ciphertext, bad base64Restore from backup; rotate the password back to the one the blob was written under
X400980VLTFMTvault crypto format failure: the stored value is not in the envelope format at all (a pre-envelope row, a botched import)shc doctor run --category vault → vault.unreadable_secrets
X503903VLTPWMvault is locked: the operation needs an unlocked vaultshc vault unlock
X503902VLTNAVvault not available with {reason} {error}: locked, or an external provider is unreachableUnlock, or check external provider health

shc doctor run --category vault runs the vault module’s diagnoses: vault.status (initialised / locked / unlocked state), vault.secrets (flags rows with an empty encrypted value, the signature of a botched import or migration), and vault.unreadable_secrets (counts rows SHC can no longer decrypt (anything not in the envelope format) and FAILS when there are any; such a row was written before v0.0.16 and its plaintext is unreachable).

External provider reads are debounced via the DebouncedVaultProvider wrapper (providers/debounce.go): identical reads within a short window are coalesced. This protects you from accidentally DOSing your paid secret manager on large install runs.

Disabling a provider while references to it still exist will cause the next render to fail with X503902VLTNAV. Either migrate the references first (shc vault set <key> <value> on a different provider and update the YAMLs), or mark the provider read-only while you migrate.


  • Never log secret values. SHC’s log module scrubs fields typed as @secret when recording events, but don’t defeat this by stuffing secrets into non-@secret variables.
  • Master password compromise ⇒ rotate immediately (shc vault rotate) and treat all previously-issued credentials as suspect.
  • Backups of /var/lib/shc contain the encrypted vault. They are still protected by the master password, but treat them with appropriate care (encrypt at rest, limit access).
  • See security.md for the full threat model.