Quota module
Layer 5. Objects. Objects A tenant owns the limits, and the HUD lists them.
The quota module enforces scoped limits on compute, storage and object
counts, plus the usage bookkeeping that backs those checks. A limit sits
on a tenant, an environment or a deployment.
Source: modules/quota/ + the enforcement hooks called from
install_flow and update_flow.
Responsibilities
Section titled “Responsibilities”- Read limits from config, scoped to a tenant, an environment or a deployment.
- Track current usage in the
usagestable (UsageModel), one row per scope. - Enforce pre-flight on install, scale and update: reject if delta would exceed the limit.
- Reconcile periodically so drift between the DB counters and reality (containers killed by the kernel, volumes deleted out-of-band) heals automatically.
- Enforce node capacity separately. A deployment that fits the
tenant quota but fits no node is still refused (
X409179PLCPND). - Expose CLI/API to operators to set limits, view usage, and force re-scans.
Limit types
Section titled “Limit types”QuotaLimits (types.go):
| Field | Unit | What it caps |
|---|---|---|
cpu_millicores | millicores | Sum of per-service CPU quotas * replicas |
memory_bytes | bytes | Sum of per-service memory quotas * replicas |
disk_bytes | bytes | Deployment data on every node + block volumes |
max_deployments | count | Top-level deployments (not vdeployments) |
max_environments | count | Distinct environment values in use |
max_apps | count | Deployments + vdeployments |
max_vdeployments | count | vdeployments only |
All limits are None-able: a None limit is “unenforced”. SHC
ships with no default quotas; nothing is limited until an
operator writes a quota: block.
Scopes
Section titled “Scopes”Quota checks traverse three scopes, most-specific first. A violation at any scope rejects the operation:
tenant always checked environment checked when an environment is named deployment checked when a deployment is namedUsage rows (UsageModel) hold one row per tenant, per environment and
per deployment. The unique constraint uq_usage_scope prevents
duplicates.
A scope is named in words, its innermost level only, in scope_name
and in every refusal, never as a path:
tenant acmeenvironment stagingdeployment blog
Resolution of limits
Section titled “Resolution of limits”Service.Limits(ctx, scope) delegates to the LimitsResolver the daemon
wires (daemon/wiring/quota_limits.go), which reads one quota.<axis>
key per cap through the scoped config cascade at the supplied scope:
quota.cpu, quota.memory, quota.disk, quota.max_deployments,
quota.max_environments, quota.max_apps, quota.max_vdeployments.
These are config-table rows, not yaml. There is no quota: block
in default.yaml. The resolver reads the config table. Writing
quota: into /etc/shc/config.yaml sets nothing. Set the caps with:
shc quota set --scope tenant -t acme --cpu 4 --memory 8g --max-deployments 10which POSTs one shc.config.set per axis at the named scope. Config
resolution and the five scope tiers are documented in
reference/configuration.
A missing key is unlimited, not zero. Each axis is a nillable
pointer; Check evaluates an axis only when its pointer is non-nil, so
an unconfigured scope is a complete no-op. A present-but-zero row is a
real cap. See
reference/quota-model-state for
what shc quota status prints in the unconfigured case and why that is
not a statement about container limits.
The per-service quota model
Section titled “The per-service quota model”Beneath those scope-level caps, every service carries a per-service
quota: one number per axis (CPU, memory). It is the input;
the compose limits and reservations the container actually runs
under are derived from it at render time:
quota = deploy.cpu_quota / deploy.memory_quota, resolved through the ordinary scoped config cascade (global → tenant → environment → deployment); when no scope sets one, the authored compose number IS the quota, the deploy.resources.limits slot, falling back to the reservations slot if only that existslimit = quota × deploy.limit_ratio (default 1.5, bounds [1, ∞))reservation = quota × deploy.reservation_ratio (default 1.0)An authored reservation is kept, and lowered to the quota when it is
above it (X200947QTACLM). Otherwise the reservation is the quota times
deploy.reservation_ratio.
Usage accounting always charges the quota, never the derived
reservation or the derived limit. Overcommit is limit_ratio: a tenant
capped at quota.cpu: "4" is charged 4 cores, and the rendered limit is
6 cores at the default ratio of 1.5.
Both ratio knobs resolve through the same scoped cascade as any other
deploy.* key. There is no app.yaml resolution tier and no node
scope for either ratio. Out-of-range or unparseable ratio values
clamp to the defaults. Install and update take --cpu-quota and
--memory-quota per service.
Parse grammar (CPU + bytes)
Section titled “Parse grammar (CPU + bytes)”modules/quota/state.go (ParseCPU / ParseBytes):
CPU cores or millicores "0.5" → 500 "1" → 1000 "2500m" → 2500
Bytes IEC-ish suffixes (case-insensitive, no decimals) "512m" → 536_870_912 "8g" → 8_589_934_592 "1t" → 1_099_511_627_776Invalid input raises X400915QTAINP.
Pre-flight enforcement
Section titled “Pre-flight enforcement”Callers use quota_service.enforce_quota(db, ctx, delta), which:
- Computes
new = current_usage + deltaper dimension. - Walks each limit; violations are collected as
QuotaCheckResult.violationsstrings. - If any non-empty, raises the first specific error code
(
X403001CPUEXC,X403002MEMEXC,X403003DSKEXC,X403004DEPEXC,X403005ENVEXC,X403006APPEXC,X403007VDPEXC).
Service.Enforce runs step 1-3 at every scope in the walk above
(EnforcementWalk), most specific first. A deployment-scoped delta must clear
the deployment, the environment and the tenant ceiling; the roll-up rows exist
for exactly this read.
The helpers in enforcement.go package this up for deployment lifecycle
callers:
| Helper | Used by |
|---|---|
Enforce (count axes) | install gate, before the QUEUED intent row |
RecordRender | install/update, inside the render |
UpdateUsageOnStackCreation | install, after the deployment row is written |
UpdateUsageOnStackDeletion | uninstall, after compose down |
UpdateUsageOnScale | the delta RecordRender and the uninstall release apply |
What gets charged
Section titled “What gets charged”The charge is the QUOTA: the one number per axis per service, times
that service’s replica count, summed over the deployment. Never the
reservation (quota x reservation_ratio), never the
limit (quota x limit_ratio, the overcommit allowance). Charging the
quota keeps overcommit equal to limit_ratio.
The figure is persisted per service in service.cpu_quota_millicores /
service.memory_quota_bytes, and a replica change is an update, whose
render charges the new total.
The cpu/memory charge cannot ride the install gate: Service.Install
writes the QUEUED intent row before ExecuteCompose renders, and there is
no per-service quota until there is a rendered compose. So counts are
charged at the gate and cpu/memory at RecordRender, inside the render and
before the deployment directory is written; an over-ceiling verdict
there is a pre-deploy refusal (nothing created, intent row deleted).
RecordRender hands an absolute total, not a delta; the adapter
converges the leaf scope onto it. An update, a same-app converge and a
retry therefore land a zero delta rather than billing the same deployment
again, and an update that lowered the caps credits the difference back.
Node capacity
Section titled “Node capacity”A replica is admitted when its reservation fits the node’s free
capacity (core/placement/place.go fits). An operator’s quota is a
request: the reservation derived from it, quota x reservation ratio,
counts at admission and at bind, while one derived from the app’s own
limit does not (SHC-2738). When no node fits, admission refuses with
X409179PLCPND (modules/fence/errors.go), naming the operator quota
behind the request if there is one, and nothing is installed. See
the resource model.
EnforceNodeCapacity compares one request with free CPU and memory on
one node (X403010NODCPU, X403011NODMEM). An axis with an unknown
total or an unknown commitment is skipped.
EnforceLargestReplicaFitsNode compares one replica with each node’s
totals and returns X403013SWMREP when none fits. Nothing outside
tests calls either helper.
Usage tracking
Section titled “Usage tracking”Every delta goes through quota_service.update_usage(...), which
upserts into UsageModel at all three scopes:
- tenant row (always)
- tenant/env row (if env given)
- deployment row (if a deployment is named)
Adds and subtracts are idempotent from the caller’s perspective; negatives on delete are symmetric.
Disk usage is separate because filesystem introspection is slow:
set_disk_usage(...) writes the measurement + disk_measured_at
timestamp directly, bypassing the delta path. See below.
Disk measurement strategy
Section titled “Disk measurement strategy”Every node measures the deployments it holds. A deployment’s data on a
node is its directory, <deployments>/<tenant>/<deployment>/<environment>,
with the volumes the render bound under its mounts/. The node answers
the named RPC quota.node_disk (modules/quota/node_disk.go) with one
row per deployment directory.
Each directory is measured on its own filesystem only, never crossing
into another one, so a block volume bound onto a mount is left out
there. modules/quota/disk.go picks the cheapest accurate method:
- xfs where the directory is the root of its own project:
xfs_quotareport of that project - zfs where the directory is a dataset’s mountpoint:
zfs list -o used - btrfs where the directory is a subvolume: its qgroup’s referenced bytes
- anything else:
du -sbx, falling back to a Go walker that stays on one device whenduis missing
A directory inside a larger project, dataset or subvolume is measured
with du: the larger figure would charge one deployment for its
neighbours.
A block volume is never measured. It is charged at the size its volume
record provisions (size_bytes), whatever the provider, since the
provider reserves all of it. A volume not yet bound to a deployment is
charged to its tenant.
The node keeps its last measurement (quota.LastNodeDisk), so a per-node
disk gauge reads the same figure without measuring again.
Backup storage
Section titled “Backup storage”A deployment’s backups count against its disk quota too (SHC-2629). The
sweep reads each deployment’s footprint from the backup module
(Footprints, ce modules/backup/footprint.go) and charges it to the
deployment, its environment and its tenant:
- In a restic repository: one
restic stats --mode raw-dataover all of the deployment’s snapshots there, the completed backup rows and the snapshot cache together. That is the bytes their blobs take in the repository, compressed, a blob two snapshots share counted once. The read runs with the deployment’s own repository credentials, and it runs again only when the deployment’s set of snapshots changes. - Outside a repository: a local
.bakarchive is charged at its recorded size, the file on disk.
A blob two deployments share is charged to each. A mirror copy of a repository is not charged: only the repository a backup row or the snapshot cache names, and the cache syncs from the primary repository.
A repository that cannot be read leaves its tenant at its last measured
figure, logged as X500909DSKSCA, the same as a node that does not
answer: the figure without those backups would be an undercount.
An operator’s shc backup create passes the same gate as an install,
RefuseMeasuredDisk, and is refused X403003DSKEXC when the
deployment, its environment or its tenant is already at its disk limit.
A scheduled backup and the safety backup that uninstall, restore and
update take are never refused: refusing them would trade the data’s
protection for a disk figure.
Background reconciliation
Section titled “Background reconciliation”Two cron jobs run on the leader:
scan_all_disk_usage: everymaintenance_disk_scan_interval(see configuration: maintenance). Asks every node forquota.node_diskat once, adds each block volume at its provisioned size and each deployment’s backup storage, and writes the tenant, environment and deployment totals. A deployment the store lists that holds no bytes on any node is written at zero. Heartbeats viarecord_heartbeat("disk_scan")soshc doctorcan flag staleness.reconcile_all_usage: everymaintenance_usage_reconcile_interval. Recomputes the deployment, vdeployment, environment and app counters directly from the deployment table. Protects against drift from crashed deploys that bypassed normal lifecycle hooks.
Neither job ever raises from the scheduler: failures are logged
as X500909DSKSCA / X500913USGREC and the next tick retries.
A node that does not answer the sweep is logged as X500909DSKSCA. The
tenants the store places on it keep their last measured figure rather
than an undercount, each logged as X503096DSKNMS, and shc quota scan
refuses for them. When the store cannot say which tenants that node
holds, no tenant is written that sweep.
The sizes an install asks for
Section titled “The sizes an install asks for”Measured usage counts what a deployment already holds. An install or
update also asks for the volumes it is about to add: the operator’s
--storage <volume>=<size>, else the app’s x-shc-volumes size. Before
any volume is provisioned and before any image is pulled, those sizes
plus measured usage must fit the disk limit of the deployment, its
environment and its tenant, or the render is refused with
X403008DSKASK, naming the scope and the in use, asked and limit
figures in the units --storage takes:
disk quota of deployment pg exceeded, 40Gi in use, 50Gi asked, 64Gi limitA block volume is provisioned at its size. A volume on the node disk has
its size charged to the disk quota and recorded on the deployment, and its
directory is capped at that size as A node-disk volume’s own cap
says. Where the node holding it cannot cap it, the size is refused with
X409311VOLQTA instead.
What the disk axis does not count
Section titled “What the disk axis does not count”A docker named volume that no render rewrote into a bind sits outside
the deployment tree, and is not counted. Backups are counted, as
Backup storage says. A boot with
no paths snapshot wires no node source, and ScanDisk refuses with
X503096DSKNMS.
Hard caps where the filesystem has a quota
Section titled “Hard caps where the filesystem has a quota”After it writes the totals, the sweep sends every node that answered
quota.node_cap: for each deployment it holds, a cap of the bytes the
deployment holds there plus the least headroom (limit less measured usage)
of its deployment, environment and tenant, never below what it already
holds. The node caps the deployment’s directories through the
filesystem’s own quota where it has one:
| Filesystem | Hard when | How |
|---|---|---|
| xfs | mounted with prjquota (or pquota), and the directory is not the root of a project SHC did not make | SHC makes the directory an xfs project of its own, the one the measurer then reads, and sets bhard |
| zfs | the directory is a dataset of its own | zfs set quota= on that dataset |
| btrfs | the directory is a subvolume and quotas are on | btrfs qgroup limit |
| anything else (ext4, a zfs or btrfs directory that is not its own dataset or subvolume, the root of an operator’s xfs project) | never | soft |
SHC numbers its xfs projects from the directory’s path, from 2^20 up. A directory that is already the root of another project is the operator’s, and SHC leaves it and its limit alone. A directory that only inherits an enclosing project becomes a project of its own, so its bytes stop counting towards the enclosing one. SHC never makes a zfs dataset or a btrfs subvolume: giving a deployment directory one of its own is how an operator asks for the hard cap there, and the cap replaces any quota already on it.
Where it is hard, a write past the cap fails with the filesystem’s own
“disk quota exceeded”. Where it is soft, usage is measured after the fact
and a refusal comes at the next install or scale (X403003DSKEXC), so a
deployment can grow past its quota between sweeps. A deployment under no
disk limit is sent a zero cap, which takes off a cap SHC set before (the
directory carries a .shc-disk-cap marker) and nothing else. The cap follows the last sweep: a
deployment held on several nodes, and several deployments under one
tenant, each get the whole headroom, so together they can pass the
tenant’s limit by up to that headroom until the next sweep refuses.
A node-disk volume’s own cap
Section titled “A node-disk volume’s own cap”A volume kept on the node disk with a size, --storage pgdata=node:20Gi,
has the size recorded in size_bytes on its row of the deployment’s
manifest.json volume_backings. A later render keeps it, and
shc update --storage pgdata=30Gi replaces it. In the same
quota.node_cap pass, after its deployment, each node caps the volume’s
directory, <deployment>/mounts/pgdata, at that size through the same
filesystem quota as the table above. On xfs SHC makes the volume an xfs
project of its own.
The size is taken only where it can be held. Before the render binds or
prepares any volume, it asks the node holding each sized node-disk volume
over app.node_disk_cap, and that node reads its own filesystem by the
table above, writing nothing. A directory not made yet is hard on an xfs
prjquota mount, which makes it a project when it is capped, and never on
zfs or btrfs, where only a dataset or subvolume of its own is. Anything
else is refused with X409311VOLQTA, naming the volume, the node and the
filesystem, so a size is never recorded and left soft. The node holding
the volume answers for itself because the leader renders every
deployment, so the node rendering may hold none of its volumes.
The manifest is on the node that renders the deployment. Each node’s
quota.node_disk answer carries the sizes the manifests it holds give,
and the sweep sends them with every holder’s cap, so a node holding the
volumes without the render caps them too. In a sweep where no node read
the manifest, a volume keeps the cap it has. The volume’s marker is
.shc-disk-cap.pgdata beside the deployment, never inside the volume,
which is the directory the container sees. A volume whose size goes has
only the cap SHC set taken off. A block volume is its provider’s size and
is not capped here. The sweep runs every hour, so a new size takes effect
at the next sweep.
An xfs project of its own, or a btrfs subvolume, keeps the volume’s bytes out of its deployment’s own figure and cap. So the measurer adds such a volume’s bytes to its deployment, and the deployment’s cap is its share less the size of each such volume, so that the deployment and its volumes together stay inside it. A zfs dataset’s quota counts its child datasets already, and anywhere else a volume is part of its deployment’s figure.
Errors
Section titled “Errors”| Code | Raised when |
|---|---|
X403001CPUEXC | a tenant, environment or deployment CPU limit exceeded |
X403002MEMEXC | memory limit exceeded |
X403003DSKEXC | disk limit exceeded |
X403004DEPEXC | deployment count limit exceeded |
X403005ENVEXC | environment count limit exceeded |
X403006APPEXC | app count limit exceeded |
X403007VDPEXC | vdeployment count limit exceeded |
X403008DSKASK | the volume sizes an install or update asks for exceed the disk limit |
X403010NODCPU | request larger than one node’s free CPU (EnforceNodeCapacity, tests only) |
X403011NODMEM | request larger than one node’s free memory (EnforceNodeCapacity, tests only) |
X403013SWMREP | one replica fits no node’s totals (EnforceLargestReplicaFitsNode, tests only) |
X400915QTAINP | invalid quota input (unparseable CPU/byte value) |
X500909DSKSCA | background disk scan failed |
X500913USGREC | usage reconcile failed |
X503096DSKNMS | disk usage is unmeasured for a scope: no node source wired, or a node holding it did not answer |
CLI / API surface
Section titled “CLI / API surface”- CLI:
shc quota {set,unset,status,scan}: see reference/cli/quota - HTTP:
modules/quota/router.goexposes CRUD at/api/v1/quota/...: see reference/api/quota
Related
Section titled “Related”- Reference: quota model: caps versus derivation: the caps-versus-derivation distinction, the defaults, and the measured state of both production estates
- Flow: install: where
enforce_stack_creation_quotafires - Flow: upgrade: where
update_usage_on_scalefires - Tenant: the primary quota scope
- Node, source of node capacity data
- Health: consumes
record_heartbeatemitted here