The daemon's settings
Every setting the daemon reads is a key, and shc config is how a key is
read and written. A key has a shipped default, and a row written with
shc config set overrides it at one scope: global, a tenant, a stack, an
environment, or one deployment. The most specific row wins. This page
turns the keys a cluster operator actually turns and reads each one back
where it takes effect. It runs on the cluster the restore page left, with
every app from the app ladder on it.
These keys are the cluster’s. The restore page left prod selected, and a
set with no --scope writes at the narrowest scope the context names, so
clear the context first. From here a row lands at global scope unless it
names another, and a command about one deployment names its tenant with
-t.
$ shc ctx --reset...Reading a key
Section titled “Reading a key”A key you never set reads its default. --show-origin says where the
value came from; ls --prefix lists the rows a section holds.
$ shc config get timeouts.drain300$ shc config get timeouts.drain --show-originKEY VALUE FOUND SOURCEtimeouts.drain 300 true default$ shc config ls --prefix backupPATH VALUE SCOPEbackup.password ref+vault://backup/password globalbackup.repo /var/backups/shc/repo globalBackup retention
Section titled “Backup retention”The nightly prune keeps as many daily, weekly, monthly and yearly backups
as these four keys say, and shc backup prune on the repositories page
applies them now. Set the four. Each write prints when it takes effect:
these are live, read by the next sweep.
$ shc config set backup.retention.daily 7takes effect on the running estate: the next reconcile sweep picks it up, no reinstall or restart neededKEY VALUE SCOPE ACTIVATIONbackup.retention.daily 7 global live$ shc config set backup.retention.weekly 4takes effect on the running estate: the next reconcile sweep picks it up, no reinstall or restart neededKEY VALUE SCOPE ACTIVATIONbackup.retention.weekly 4 global live$ shc config set backup.retention.monthly 6takes effect on the running estate: the next reconcile sweep picks it up, no reinstall or restart neededKEY VALUE SCOPE ACTIVATIONbackup.retention.monthly 6 global live$ shc config set backup.retention.yearly 1takes effect on the running estate: the next reconcile sweep picks it up, no reinstall or restart neededKEY VALUE SCOPE ACTIVATIONbackup.retention.yearly 1 global live$ shc config get backup.retention.daily7A row at a narrower scope beats the global one for what it covers. Keep
three daily backups of the prod tenant’s deployments: --scope tenant
with the tenant named. Read with the tenant named and its row answers;
read without it and the global row does. --scope on get reads
exactly one tier, and on ls lists it.
$ shc config set backup.retention.daily 3 --scope tenant -t prodtakes effect on the running estate: the next reconcile sweep picks it up, no reinstall or restart neededKEY VALUE SCOPE ACTIVATIONbackup.retention.daily 3 tenant live$ shc config get backup.retention.daily -t prod3$ shc config get backup.retention.daily...7...$ shc config get backup.retention.daily --scope tenant -t prod3$ shc config get backup.retention.daily --scope global7$ shc config ls --scope tenant -t prodPATH VALUE SCOPEbackup.retention.daily 3 tenantthis listing covers the tenant scope only: 22 further config rows are set at narrower scopes in tenant master and are NOT shown above$ shc config ls --prefix backup.retention -t prod...The read without a tenant still answers 7, and it says on stderr that a
row at tenant scope in prod was passed over, with the command that lists
it. ls --all lists every row of a key at every scope, each with the
tenant, stack and environment it is set at.
$ shc config ls --all --prefix backup.retention.dailyPATH VALUE SCOPE TENANT...Not every key is read at every scope. Retention is read at the global,
tenant and stack tiers and nowhere else, so a row at the environment
tier would be stored and consulted by nothing; shc config set refuses
it and names the tiers that work.
$ shc config set backup.retention.daily 3 --scope environment -t prod -e stagingshc: [X400066CFGSCP] config key backup.retention.daily is never read at environment scope, nothing would consult this row, set it at stack, tenant, global insteadunset removes one row at one scope and nothing else; the global value
shows through again.
$ shc config unset backup.retention.daily --scope tenant -t prod...$ shc config get backup.retention.daily -t prod7Timeouts
Section titled “Timeouts”A timeout is a deadline in seconds for one kind of operation, cluster
wide. timeouts.drain is the budget for each relocation a drain makes,
not for the whole drain: shc node drain --timeout bounds a run. The
three backup budgets nest: timeouts.backup bounds the restic call,
timeouts.backup_converter one app’s converter, timeouts.backup_stack
the whole capture of one deployment. timeouts.quiesce bounds taking
the freeze a backup asks an app for and timeouts.quiesce_hold keeping
it; past the hold the backup stops the app rather than hold a freeze of
unknown age. A row is a plain number and reads back as one.
$ shc config set timeouts.drain 120KEY VALUE SCOPEtimeouts.drain 120 global$ shc config get timeouts.drain --show-originKEY VALUE FOUND SOURCE SCOPEtimeouts.drain 120 true row global$ shc config set timeouts.backup 3600KEY VALUE SCOPEtimeouts.backup 3600 global$ shc config set timeouts.backup_converter 10800KEY VALUE SCOPEtimeouts.backup_converter 10800 global$ shc config set timeouts.backup_stack 43200KEY VALUE SCOPEtimeouts.backup_stack 43200 global$ shc config set timeouts.quiesce 60KEY VALUE SCOPEtimeouts.quiesce 60 global$ shc config set timeouts.quiesce_hold 600KEY VALUE SCOPEtimeouts.quiesce_hold 600 global$ shc config set timeouts.container_stop 20KEY VALUE SCOPEtimeouts.container_stop 20 global$ shc config set timeouts.compose_converge 1200KEY VALUE SCOPEtimeouts.compose_converge 1200 global$ shc config set timeouts.exec_idle 3600KEY VALUE SCOPEtimeouts.exec_idle 3600 global$ shc config set timeouts.log_idle 600KEY VALUE SCOPEtimeouts.log_idle 600 global$ shc config ls --prefix timeouts...The other timeouts bound the daemon’s own calls: its Keycloak and database connections, the store coming up and shutting down, a mesh ping, a node enrolling, a repository clone and fetch, and how long an exec waits after its stop signal before the kill. Their defaults fit a lab and they are set the same way.
$ shc config set timeouts.keycloak 20$ shc config set timeouts.db_connect 20$ shc config set timeouts.rqlite_ready 120$ shc config set timeouts.rqlite_shutdown 20$ shc config set timeouts.mesh_ping 20$ shc config set timeouts.enroll 120$ shc config set timeouts.git_clone 1200$ shc config set timeouts.git_fetch 600$ shc config set timeouts.exec_kill_wait 10A trailing ! on the key removes the row, the same as unset. Put the
drain budget back and the default reads again.
$ shc config set timeouts.drain!...$ shc config get timeouts.drain300The scheduler
Section titled “The scheduler”shc constraint explain prints two of the scheduler’s keys for a
deployment: its priority on the first line and, at the end, how long its
replicas wait after their node’s lease expires before they move. Read
web.
$ shc constraint explain -s web -t proddeployment prod/web/prod movability free priority normalrunning on home03status running
DERIVED CONSTRAINTS limit condition ready only ready nodes take new work keeps far01 home01 home02 home03 limit architecture images the replica's images must have a manifest for the node's architecture keeps far01 home01 home02 home03 limit fit room the replica needs capacity for its CPU and memory reservations; home01: cpu: past capacity, 3100m committed of 2000m; home02: cpu: past capacity, 5350m committed of 2000m removes home01 home02; keeps far01 home03
eligible far01 home03
RELOCATION far01: possible shc update -s web -e prod --where 'node=far01'
failover 1m0s after its node's lease expires: a new assignment inside its eligible setBoth keys are read through the scope cascade, so a tenant, a stack or
one deployment can be set apart from the rest. scheduler.priority orders the scheduler’s
work, high first, and never preempts; scheduler.grace.stateless is the
wait for a replica with no volume, scheduler.grace.member for one with
a volume or in a group, which never waits less than a minute. Raise
the priority and lengthen the grace for everything in prod, at the
tenant tier, and explain reads both on web.
$ shc config set scheduler.priority high --scope tenant -t prodKEY VALUE SCOPEscheduler.priority high tenant$ shc config set scheduler.grace.stateless 90s --scope tenant -t prodKEY VALUE SCOPEscheduler.grace.stateless 90s tenant$ shc constraint explain -s web -t proddeployment prod/web/prod movability free priority highrunning on home03status running
DERIVED CONSTRAINTS limit condition ready only ready nodes take new work keeps far01 home01 home02 home03 limit architecture images the replica's images must have a manifest for the node's architecture keeps far01 home01 home02 home03 limit fit room the replica needs capacity for its CPU and memory reservations; home01: cpu: past capacity, 3100m committed of 2000m; home02: cpu: past capacity, 5350m committed of 2000m removes home01 home02; keeps far01 home03
eligible far01 home03
RELOCATION far01: possible shc update -s web -e prod --where 'node=far01'
failover 1m30s after its node's lease expires: a new assignment inside its eligible setThe member grace and the move rate are global. scheduler.moves_per_minute
caps the automatic moves the cluster starts inside any minute, a
reschedule, a failover or a drain’s moves alike, and a manual move is
never counted. It is read when the daemon starts, so a change waits for
the next restart; the upgrade page restarts every node.
$ shc config set scheduler.grace.member 10mKEY VALUE SCOPEscheduler.grace.member 10m global$ shc config get scheduler.grace.member10m$ shc config set scheduler.moves_per_minute 3KEY VALUE SCOPEscheduler.moves_per_minute 3 global$ shc config get scheduler.moves_per_minute3Take the two tenant rows away and web reads as it did.
$ shc config unset scheduler.priority --scope tenant -t prod...$ shc config unset scheduler.grace.stateless --scope tenant -t prod...$ shc constraint explain -s web -t proddeployment prod/web/prod movability free priority normalrunning on home03status running
DERIVED CONSTRAINTS limit condition ready only ready nodes take new work keeps far01 home01 home02 home03 limit architecture images the replica's images must have a manifest for the node's architecture keeps far01 home01 home02 home03 limit fit room the replica needs capacity for its CPU and memory reservations; home01: cpu: past capacity, 3100m committed of 2000m; home02: cpu: past capacity, 5350m committed of 2000m removes home01 home02; keeps far01 home03
eligible far01 home03
RELOCATION far01: possible shc update -s web -e prod --where 'node=far01'
failover 1m0s after its node's lease expires: a new assignment inside its eligible setMaintenance
Section titled “Maintenance”The maintenance keys bound what the control plane keeps. Events are
kept event_retention_days; a code its module declares chatter lives
event_retention_short_days, one it declares a significant transition
event_retention_long_days, and both are tiers over the first number.
event_max_rows caps the table by count. A row still waiting to be
broadcast to a peer is protected from the pruner for
event_broadcast_grace_days; event_pending_max and
event_pending_ttl_seconds bound how many such rows may wait and for
how long. A node prunes images once its disk passes
disk_auto_prune_threshold percent, exited job containers are reaped
after stale_jobs_retention_hours, and the leader keeps
control_plane_snapshot_retention nightly snapshots of the store. The
four containment keys cap what one event row may cost and how many
identical rows one code may write inside a window. All are global and
live.
$ shc config get maintenance.event_retention_days14$ shc config set maintenance.event_retention_days 7...$ shc config set maintenance.event_retention_short_days 1...$ shc config set maintenance.event_retention_long_days 30...$ shc config set maintenance.event_max_rows 200000...$ shc config set maintenance.event_broadcast_grace_days 2...$ shc config set maintenance.event_pending_max 5000...$ shc config set maintenance.event_pending_ttl_seconds 1800...$ shc config set maintenance.disk_auto_prune_threshold 70...$ shc config set maintenance.stale_jobs_retention_hours 12...$ shc config set maintenance.control_plane_snapshot_retention 14...$ shc config set maintenance.event_payload_max_bytes 16384...$ shc config set maintenance.event_message_max_bytes 4096...$ shc config set maintenance.event_coalesce_window_seconds 30...$ shc config set maintenance.event_coalesce_max_per_window 5...$ shc config get maintenance.event_retention_days7$ shc config ls --prefix maintenance...shc db purge on the upgrade page reports what the retention rules
would take; with the days cut to seven it reports more than it did.
$ shc db purge...Nine more maintenance keys stay as shipped on this ladder. Six of
them bound the audit records: the audit table is kept 365 days and the
audit events 90, neither is capped by count, and identical audit rows
are never coalesced into one with a count. Shortening the days, capping
the rows or coalescing is an estate owner’s ruling, because deleted
audit records are gone and the audit chain shows the cut forever. The
last three fold the vault’s internal reads into one audit row per
window; zero restores a row per read.
$ shc config set maintenance.audit_retention_days 180$ shc config set maintenance.audit_max_rows 1000000$ shc config set maintenance.event_audit_retention_days 90$ shc config set maintenance.event_audit_max_rows 1000000$ shc config set maintenance.audit_coalesce_window_seconds 60$ shc config set maintenance.audit_coalesce_max_per_window 10$ shc config set maintenance.vault_read_audit_window_seconds 0$ shc config set maintenance.vault_read_audit_max_keys 2048$ shc config set maintenance.vault_read_audit_steady_attest_seconds 3600Put the event retention back to the shipped fourteen days.
$ shc config unset maintenance.event_retention_days...$ shc config get maintenance.event_retention_days14The log level
Section titled “The log level”log.level has no shipped row: the daemon starts at info unless its
environment says otherwise. A row written here is picked up by every
node’s running daemon, which logs the change. Raise it, read the
daemon’s log, and take the row away again.
$ shc config set log.level debugKEY VALUE SCOPElog.level debug global$ shc config get log.leveldebug$ shc cluster logs home01 -S shc -n 5...$ shc config set log.level!...Health, placement and the queues
Section titled “Health, placement and the queues”The self-heal reconcile runs on the leader. health.reconcile_enabled
switches it, health.check_infrastructure and
health.check_docker_networks switch two of its passes, and
health.adopt_keycloak_realms is whether a realm found in the system
Keycloak that is no tenant becomes one. Each is read on the next pass.
deploy.placement_scoring is whether an install ranks every node or
only the one that received it; it is read at each install.
concurrency.queue_max_depth is the ceiling of the daemon’s work
queues, read live. timing.tick_interval is how often each node
measures its containers, read when the daemon starts, and
timing.slow_poll the pace of its slow loops, read on the next cycle.
$ shc config set health.reconcile_enabled trueKEY VALUE SCOPEhealth.reconcile_enabled true global$ shc config set health.check_infrastructure falseKEY VALUE SCOPEhealth.check_infrastructure false global$ shc config set health.check_docker_networks falseKEY VALUE SCOPEhealth.check_docker_networks false global$ shc config set health.adopt_keycloak_realms falseKEY VALUE SCOPEhealth.adopt_keycloak_realms false global$ shc config ls --prefix health...$ shc config set deploy.placement_scoring offKEY VALUE SCOPEdeploy.placement_scoring off global$ shc config get deploy.placement_scoringoff$ shc config set concurrency.queue_max_depth 2000KEY VALUE SCOPEconcurrency.queue_max_depth 2000 global$ shc config get concurrency.queue_max_depth2000$ shc config set timing.tick_interval 'every 10 seconds'KEY VALUE SCOPEtiming.tick_interval every 10 seconds global$ shc config get timing.tick_intervalevery 10 seconds$ shc config set timing.slow_poll 2KEY VALUE SCOPEtiming.slow_poll 2 global$ shc config get timing.slow_poll2The deploy quota keys are the other activation class. A limit ratio,
a reservation ratio or a quota is baked into a deployment’s render when
it is installed; a running container keeps the figure it was installed
with until shc update renders it again, and the write says so.
An app’s deploy.resources.limits is read as the deployment’s quota when
nothing higher sets one, and the container runs with quota x
deploy.limit_ratio (1.5 by default), so gitlab’s cpus: "2.0" runs with a
3.0 CPU cap. shc inspect <stack> shows the rendered figure.
shc config set deploy.limit_ratio 1, then shc update <stack>, makes the
cap the authored number when that authored limit supplies the quota.
$ shc config set deploy.limit_ratio 1.2read only while a deployment is rendered: a running container keeps the value it was installed with, run `shc update <stack>` to apply itresource figures are baked into the rendered compose: run `shc update <stack>` to re-render an existing deployment at the new value, and `shc inspect <stack>` to read the limits its rendered compose gives each containerKEY VALUE SCOPE ACTIVATIONdeploy.limit_ratio 1.2 global install$ shc config get deploy.limit_ratio --show-originKEY VALUE FOUND SOURCE SCOPEdeploy.limit_ratio 1.2 true row globalThe rest of that family are set the same way: a CPU and a memory quota per deployment, the ratio of a declared limit that becomes the reservation the scheduler counts, and that ratio per axis. This ladder renders nothing again, so they are not typed.
$ shc config set deploy.cpu_quota 2$ shc config set deploy.memory_quota 4g$ shc config set deploy.reservation_ratio 0.5$ shc config set deploy.cpu_reservation_ratio 0.25$ shc config set deploy.memory_reservation_ratio 1.0Take the rows this section wrote away. The four health keys, the scoring, the queue depth and the two timing keys go back to their defaults; the limit ratio stays until the next render.
$ shc config set health.reconcile_enabled!...$ shc config set health.check_infrastructure!...$ shc config set health.check_docker_networks!...$ shc config set health.adopt_keycloak_realms!...$ shc config set deploy.placement_scoring!...$ shc config set concurrency.queue_max_depth!...$ shc config set timing.tick_interval!...$ shc config set timing.slow_poll!...$ shc config set deploy.limit_ratio!...$ shc config ls --prefix health...The keys the daemon reads for itself
Section titled “The keys the daemon reads for itself”Some keys are the daemon’s own, and shc config set is not how they
are set. dev reads false and shc config set refuses it.
$ shc config get devfalsedev and the two collector cardinality caps are read once, from
/etc/shc/config.yaml on the node, when the daemon starts; a row in
the store is refused rather than stored and ignored.
$ shc config set dev true$ shc config set collector.max_cardinality_per_instrument 2000$ shc config set collector.diagnostic_cardinality 200The log format, the store’s path or URL, the internal listener’s bind
address and whether it is on, the token the nodes present to one
another and the default environment are read once while the daemon
starts, some before the store is even open, from its environment on the
node: SHC_LOG_FORMAT, SHC_DATABASE_PATH, SHC_INTERNAL_TOKEN,
SHC_ENVIRONMENT. A row written here is stored and read back, and the
running daemon keeps what it started with.
$ shc config set log.format json$ shc config set database.path /var/lib/shc/shc.db$ shc config set database.url sqlite:///var/lib/shc/shc.db$ shc config set server.tcp.enabled true$ shc config set server.tcp.bind 0.0.0.0:4001$ shc config set internal_api.token P@ssw0rd$ shc config set environment stagingcrons and modules are sections, not keys: each member is named in
code and set by its own path. A cron’s schedule is overridden under its
name, as shc cron ls on the upgrade page lists them; a platform
module’s host, path or variables under the module’s name, as the
observability page sets the OpenObserve password. This ladder keeps the
shipped schedules and module set.
$ shc config set crons.db_snapshot_cron '0 2 * * *'$ shc config get crons$ shc config set modules.keycloak.host auth.example.com$ shc config get modulesThe five license keys steer the licence the licence page installed
from a file: whether a trial is accepted without the command, whether
the daemon re-checks the licence with the broker, whether the DNS
record carries it, under which domain, and license.offline for an
estate that never dials out. The licence here came from a file and
stays as it is.
$ shc config set license.accept_trial true$ shc config set license.auto_check false$ shc config set license.dns_enabled false$ shc config set license.domain learn.example$ shc config set license.offline trueThe three cluster keys belong to an estate that shares a DNS zone
with another. cluster.dns_authority set to standby stops this
cluster publishing any managed record until it is made authoritative
at the handover; cluster.dns_takeover is whether it may take over
records another cluster wrote; cluster.generation is the generation
the records carry, and the platform writes it. One cluster owns this
ladder’s zone.
$ shc config set cluster.dns_authority standby$ shc config set cluster.dns_takeover allow$ shc config set cluster.generation g2The backup repository on this ladder is a directory. A repository in
S3, Azure, Backblaze or Google Cloud Storage takes the account’s
credentials from the backup keys, whose secrets shc config get
prints redacted; a ref+vault:// reference, as backup.password is,
keeps the secret in the vault.
$ shc config set backup.aws_access_key_id AKIAEXAMPLE$ shc config set backup.aws_secret_access_key P@ssw0rd$ shc config set backup.azure_account_name learn$ shc config set backup.azure_account_key P@ssw0rd$ shc config set backup.b2_account_id 000example$ shc config set backup.b2_account_key P@ssw0rd$ shc config set backup.google_application_credentials /etc/shc/gcs.jsonThe cluster is as the restore page left it, with the retention, the
timeouts, the maintenance bounds and the two global scheduler rows this
page set. shc config ls lists every row.
$ shc config ls...That is the whole ladder.