Skip to content
SHC Docs

Quota module

Layer 5. Objects. Objects A tenant owns the limits, and the HUD lists them.

The quota module enforces scoped limits on compute, storage and object counts, plus the usage bookkeeping that backs those checks. A limit sits on a tenant, an environment or a deployment.

Source: modules/quota/ + the enforcement hooks called from install_flow and update_flow.

  1. Read limits from config, scoped to a tenant, an environment or a deployment.
  2. Track current usage in the usages table (UsageModel), one row per scope.
  3. Enforce pre-flight on install, scale and update: reject if delta would exceed the limit.
  4. Reconcile periodically so drift between the DB counters and reality (containers killed by the kernel, volumes deleted out-of-band) heals automatically.
  5. Enforce node capacity separately. A deployment that fits the tenant quota but fits no node is still refused (X409179PLCPND).
  6. Expose CLI/API to operators to set limits, view usage, and force re-scans.

QuotaLimits (types.go):

FieldUnitWhat it caps
cpu_millicoresmillicoresSum of per-service CPU quotas * replicas
memory_bytesbytesSum of per-service memory quotas * replicas
disk_bytesbytesDeployment data on every node + block volumes
max_deploymentscountTop-level deployments (not vdeployments)
max_environmentscountDistinct environment values in use
max_appscountDeployments + vdeployments
max_vdeploymentscountvdeployments only

All limits are None-able: a None limit is “unenforced”. SHC ships with no default quotas; nothing is limited until an operator writes a quota: block.

Quota checks traverse three scopes, most-specific first. A violation at any scope rejects the operation:

tenant always checked
environment checked when an environment is named
deployment checked when a deployment is named

Usage rows (UsageModel) hold one row per tenant, per environment and per deployment. The unique constraint uq_usage_scope prevents duplicates.

A scope is named in words, its innermost level only, in scope_name and in every refusal, never as a path:

  • tenant acme
  • environment staging
  • deployment blog

Service.Limits(ctx, scope) delegates to the LimitsResolver the daemon wires (daemon/wiring/quota_limits.go), which reads one quota.<axis> key per cap through the scoped config cascade at the supplied scope: quota.cpu, quota.memory, quota.disk, quota.max_deployments, quota.max_environments, quota.max_apps, quota.max_vdeployments.

These are config-table rows, not yaml. There is no quota: block in default.yaml. The resolver reads the config table. Writing quota: into /etc/shc/config.yaml sets nothing. Set the caps with:

Terminal window
shc quota set --scope tenant -t acme --cpu 4 --memory 8g --max-deployments 10

which POSTs one shc.config.set per axis at the named scope. Config resolution and the five scope tiers are documented in reference/configuration.

A missing key is unlimited, not zero. Each axis is a nillable pointer; Check evaluates an axis only when its pointer is non-nil, so an unconfigured scope is a complete no-op. A present-but-zero row is a real cap. See reference/quota-model-state for what shc quota status prints in the unconfigured case and why that is not a statement about container limits.

Beneath those scope-level caps, every service carries a per-service quota: one number per axis (CPU, memory). It is the input; the compose limits and reservations the container actually runs under are derived from it at render time:

quota = deploy.cpu_quota / deploy.memory_quota, resolved through
the ordinary scoped config cascade
(global → tenant → environment → deployment);
when no scope sets one, the authored compose number IS
the quota, the deploy.resources.limits slot, falling
back to the reservations slot if only that exists
limit = quota × deploy.limit_ratio (default 1.5, bounds [1, ∞))
reservation = quota × deploy.reservation_ratio (default 1.0)

An authored reservation is kept, and lowered to the quota when it is above it (X200947QTACLM). Otherwise the reservation is the quota times deploy.reservation_ratio.

Usage accounting always charges the quota, never the derived reservation or the derived limit. Overcommit is limit_ratio: a tenant capped at quota.cpu: "4" is charged 4 cores, and the rendered limit is 6 cores at the default ratio of 1.5.

Both ratio knobs resolve through the same scoped cascade as any other deploy.* key. There is no app.yaml resolution tier and no node scope for either ratio. Out-of-range or unparseable ratio values clamp to the defaults. Install and update take --cpu-quota and --memory-quota per service.

modules/quota/state.go (ParseCPU / ParseBytes):

CPU cores or millicores
"0.5" → 500
"1" → 1000
"2500m" → 2500
Bytes IEC-ish suffixes (case-insensitive, no decimals)
"512m" → 536_870_912
"8g" → 8_589_934_592
"1t" → 1_099_511_627_776

Invalid input raises X400915QTAINP.

Callers use quota_service.enforce_quota(db, ctx, delta), which:

  1. Computes new = current_usage + delta per dimension.
  2. Walks each limit; violations are collected as QuotaCheckResult.violations strings.
  3. If any non-empty, raises the first specific error code (X403001CPUEXC, X403002MEMEXC, X403003DSKEXC, X403004DEPEXC, X403005ENVEXC, X403006APPEXC, X403007VDPEXC).

Service.Enforce runs step 1-3 at every scope in the walk above (EnforcementWalk), most specific first. A deployment-scoped delta must clear the deployment, the environment and the tenant ceiling; the roll-up rows exist for exactly this read.

The helpers in enforcement.go package this up for deployment lifecycle callers:

HelperUsed by
Enforce (count axes)install gate, before the QUEUED intent row
RecordRenderinstall/update, inside the render
UpdateUsageOnStackCreationinstall, after the deployment row is written
UpdateUsageOnStackDeletionuninstall, after compose down
UpdateUsageOnScalethe delta RecordRender and the uninstall release apply

The charge is the QUOTA: the one number per axis per service, times that service’s replica count, summed over the deployment. Never the reservation (quota x reservation_ratio), never the limit (quota x limit_ratio, the overcommit allowance). Charging the quota keeps overcommit equal to limit_ratio.

The figure is persisted per service in service.cpu_quota_millicores / service.memory_quota_bytes, and a replica change is an update, whose render charges the new total.

The cpu/memory charge cannot ride the install gate: Service.Install writes the QUEUED intent row before ExecuteCompose renders, and there is no per-service quota until there is a rendered compose. So counts are charged at the gate and cpu/memory at RecordRender, inside the render and before the deployment directory is written; an over-ceiling verdict there is a pre-deploy refusal (nothing created, intent row deleted).

RecordRender hands an absolute total, not a delta; the adapter converges the leaf scope onto it. An update, a same-app converge and a retry therefore land a zero delta rather than billing the same deployment again, and an update that lowered the caps credits the difference back.

A replica is admitted when its reservation fits the node’s free capacity (core/placement/place.go fits). An operator’s quota is a request: the reservation derived from it, quota x reservation ratio, counts at admission and at bind, while one derived from the app’s own limit does not (SHC-2738). When no node fits, admission refuses with X409179PLCPND (modules/fence/errors.go), naming the operator quota behind the request if there is one, and nothing is installed. See the resource model.

EnforceNodeCapacity compares one request with free CPU and memory on one node (X403010NODCPU, X403011NODMEM). An axis with an unknown total or an unknown commitment is skipped. EnforceLargestReplicaFitsNode compares one replica with each node’s totals and returns X403013SWMREP when none fits. Nothing outside tests calls either helper.

Every delta goes through quota_service.update_usage(...), which upserts into UsageModel at all three scopes:

  • tenant row (always)
  • tenant/env row (if env given)
  • deployment row (if a deployment is named)

Adds and subtracts are idempotent from the caller’s perspective; negatives on delete are symmetric.

Disk usage is separate because filesystem introspection is slow: set_disk_usage(...) writes the measurement + disk_measured_at timestamp directly, bypassing the delta path. See below.

Every node measures the deployments it holds. A deployment’s data on a node is its directory, <deployments>/<tenant>/<deployment>/<environment>, with the volumes the render bound under its mounts/. The node answers the named RPC quota.node_disk (modules/quota/node_disk.go) with one row per deployment directory.

Each directory is measured on its own filesystem only, never crossing into another one, so a block volume bound onto a mount is left out there. modules/quota/disk.go picks the cheapest accurate method:

  1. xfs where the directory is the root of its own project: xfs_quota report of that project
  2. zfs where the directory is a dataset’s mountpoint: zfs list -o used
  3. btrfs where the directory is a subvolume: its qgroup’s referenced bytes
  4. anything else: du -sbx, falling back to a Go walker that stays on one device when du is missing

A directory inside a larger project, dataset or subvolume is measured with du: the larger figure would charge one deployment for its neighbours.

A block volume is never measured. It is charged at the size its volume record provisions (size_bytes), whatever the provider, since the provider reserves all of it. A volume not yet bound to a deployment is charged to its tenant.

The node keeps its last measurement (quota.LastNodeDisk), so a per-node disk gauge reads the same figure without measuring again.

A deployment’s backups count against its disk quota too (SHC-2629). The sweep reads each deployment’s footprint from the backup module (Footprints, ce modules/backup/footprint.go) and charges it to the deployment, its environment and its tenant:

  • In a restic repository: one restic stats --mode raw-data over all of the deployment’s snapshots there, the completed backup rows and the snapshot cache together. That is the bytes their blobs take in the repository, compressed, a blob two snapshots share counted once. The read runs with the deployment’s own repository credentials, and it runs again only when the deployment’s set of snapshots changes.
  • Outside a repository: a local .bak archive is charged at its recorded size, the file on disk.

A blob two deployments share is charged to each. A mirror copy of a repository is not charged: only the repository a backup row or the snapshot cache names, and the cache syncs from the primary repository.

A repository that cannot be read leaves its tenant at its last measured figure, logged as X500909DSKSCA, the same as a node that does not answer: the figure without those backups would be an undercount.

An operator’s shc backup create passes the same gate as an install, RefuseMeasuredDisk, and is refused X403003DSKEXC when the deployment, its environment or its tenant is already at its disk limit. A scheduled backup and the safety backup that uninstall, restore and update take are never refused: refusing them would trade the data’s protection for a disk figure.

Two cron jobs run on the leader:

  • scan_all_disk_usage: every maintenance_disk_scan_interval (see configuration: maintenance). Asks every node for quota.node_disk at once, adds each block volume at its provisioned size and each deployment’s backup storage, and writes the tenant, environment and deployment totals. A deployment the store lists that holds no bytes on any node is written at zero. Heartbeats via record_heartbeat("disk_scan") so shc doctor can flag staleness.
  • reconcile_all_usage: every maintenance_usage_reconcile_interval. Recomputes the deployment, vdeployment, environment and app counters directly from the deployment table. Protects against drift from crashed deploys that bypassed normal lifecycle hooks.

Neither job ever raises from the scheduler: failures are logged as X500909DSKSCA / X500913USGREC and the next tick retries.

A node that does not answer the sweep is logged as X500909DSKSCA. The tenants the store places on it keep their last measured figure rather than an undercount, each logged as X503096DSKNMS, and shc quota scan refuses for them. When the store cannot say which tenants that node holds, no tenant is written that sweep.

Measured usage counts what a deployment already holds. An install or update also asks for the volumes it is about to add: the operator’s --storage <volume>=<size>, else the app’s x-shc-volumes size. Before any volume is provisioned and before any image is pulled, those sizes plus measured usage must fit the disk limit of the deployment, its environment and its tenant, or the render is refused with X403008DSKASK, naming the scope and the in use, asked and limit figures in the units --storage takes:

disk quota of deployment pg exceeded, 40Gi in use, 50Gi asked, 64Gi limit

A block volume is provisioned at its size. A volume on the node disk has its size charged to the disk quota and recorded on the deployment, and its directory is capped at that size as A node-disk volume’s own cap says. Where the node holding it cannot cap it, the size is refused with X409311VOLQTA instead.

A docker named volume that no render rewrote into a bind sits outside the deployment tree, and is not counted. Backups are counted, as Backup storage says. A boot with no paths snapshot wires no node source, and ScanDisk refuses with X503096DSKNMS.

Hard caps where the filesystem has a quota

Section titled “Hard caps where the filesystem has a quota”

After it writes the totals, the sweep sends every node that answered quota.node_cap: for each deployment it holds, a cap of the bytes the deployment holds there plus the least headroom (limit less measured usage) of its deployment, environment and tenant, never below what it already holds. The node caps the deployment’s directories through the filesystem’s own quota where it has one:

FilesystemHard whenHow
xfsmounted with prjquota (or pquota), and the directory is not the root of a project SHC did not makeSHC makes the directory an xfs project of its own, the one the measurer then reads, and sets bhard
zfsthe directory is a dataset of its ownzfs set quota= on that dataset
btrfsthe directory is a subvolume and quotas are onbtrfs qgroup limit
anything else (ext4, a zfs or btrfs directory that is not its own dataset or subvolume, the root of an operator’s xfs project)neversoft

SHC numbers its xfs projects from the directory’s path, from 2^20 up. A directory that is already the root of another project is the operator’s, and SHC leaves it and its limit alone. A directory that only inherits an enclosing project becomes a project of its own, so its bytes stop counting towards the enclosing one. SHC never makes a zfs dataset or a btrfs subvolume: giving a deployment directory one of its own is how an operator asks for the hard cap there, and the cap replaces any quota already on it.

Where it is hard, a write past the cap fails with the filesystem’s own “disk quota exceeded”. Where it is soft, usage is measured after the fact and a refusal comes at the next install or scale (X403003DSKEXC), so a deployment can grow past its quota between sweeps. A deployment under no disk limit is sent a zero cap, which takes off a cap SHC set before (the directory carries a .shc-disk-cap marker) and nothing else. The cap follows the last sweep: a deployment held on several nodes, and several deployments under one tenant, each get the whole headroom, so together they can pass the tenant’s limit by up to that headroom until the next sweep refuses.

A volume kept on the node disk with a size, --storage pgdata=node:20Gi, has the size recorded in size_bytes on its row of the deployment’s manifest.json volume_backings. A later render keeps it, and shc update --storage pgdata=30Gi replaces it. In the same quota.node_cap pass, after its deployment, each node caps the volume’s directory, <deployment>/mounts/pgdata, at that size through the same filesystem quota as the table above. On xfs SHC makes the volume an xfs project of its own.

The size is taken only where it can be held. Before the render binds or prepares any volume, it asks the node holding each sized node-disk volume over app.node_disk_cap, and that node reads its own filesystem by the table above, writing nothing. A directory not made yet is hard on an xfs prjquota mount, which makes it a project when it is capped, and never on zfs or btrfs, where only a dataset or subvolume of its own is. Anything else is refused with X409311VOLQTA, naming the volume, the node and the filesystem, so a size is never recorded and left soft. The node holding the volume answers for itself because the leader renders every deployment, so the node rendering may hold none of its volumes.

The manifest is on the node that renders the deployment. Each node’s quota.node_disk answer carries the sizes the manifests it holds give, and the sweep sends them with every holder’s cap, so a node holding the volumes without the render caps them too. In a sweep where no node read the manifest, a volume keeps the cap it has. The volume’s marker is .shc-disk-cap.pgdata beside the deployment, never inside the volume, which is the directory the container sees. A volume whose size goes has only the cap SHC set taken off. A block volume is its provider’s size and is not capped here. The sweep runs every hour, so a new size takes effect at the next sweep.

An xfs project of its own, or a btrfs subvolume, keeps the volume’s bytes out of its deployment’s own figure and cap. So the measurer adds such a volume’s bytes to its deployment, and the deployment’s cap is its share less the size of each such volume, so that the deployment and its volumes together stay inside it. A zfs dataset’s quota counts its child datasets already, and anywhere else a volume is part of its deployment’s figure.

CodeRaised when
X403001CPUEXCa tenant, environment or deployment CPU limit exceeded
X403002MEMEXCmemory limit exceeded
X403003DSKEXCdisk limit exceeded
X403004DEPEXCdeployment count limit exceeded
X403005ENVEXCenvironment count limit exceeded
X403006APPEXCapp count limit exceeded
X403007VDPEXCvdeployment count limit exceeded
X403008DSKASKthe volume sizes an install or update asks for exceed the disk limit
X403010NODCPUrequest larger than one node’s free CPU (EnforceNodeCapacity, tests only)
X403011NODMEMrequest larger than one node’s free memory (EnforceNodeCapacity, tests only)
X403013SWMREPone replica fits no node’s totals (EnforceLargestReplicaFitsNode, tests only)
X400915QTAINPinvalid quota input (unparseable CPU/byte value)
X500909DSKSCAbackground disk scan failed
X500913USGRECusage reconcile failed
X503096DSKNMSdisk usage is unmeasured for a scope: no node source wired, or a node holding it did not answer