The incus runtime (design)
Status: open design (SHC-1199). The executor is linked into shcd, and three items section 9 once listed are built: node-agent restarts (ce
modules/nodeagentstarts the one instance), the per-tenant fabric bridge (a kernel bridge per tenant slot, SHC-1322, and a routed fabric NIC, SHC-1227, not section 4’sshc-fabricIncus bridge) and image cache pruning (cecore/executor/incus/image_prune.go). Not built: the foreign-instance fact and thehypervisor-not-quoratecondition (section 5). A restore, clone, move or rename onto incus is taken (SHC-3155). Clay ruled on 2026-09-30 and 2026-10-01 that SHC’s runtimes are compose runtimes named for the engine (docker, podman, incus, apple), that Incus is driven through lxc/incus-compose, that no Docker API server and no Proxmox executor get built, and that each Incus host is its own SHC node, never an Incus cluster (memories: shc-runtimes-docker-podman-incus, shc-on-proxmox-shc-decides-proxmox-executes). SHC decides and Incus executes: Incus never restarts, moves or heals an SHC instance on its own. This note is the contract for cecore/executor/incus. It does not edit the simplification or scheduler proposals; the deltas it owes them are in section 10.
Measured against Incus 7.5.1 (Zabbly stable, Debian 13, kernel
6.12.111+deb13-cloud-amd64) and incus-compose built from main at
3bec850b (2026-10-01).
1. What incus-compose makes
Section titled “1. What incus-compose makes”One compose project becomes one Incus project, named from the compose
name (lowercased, anything else a hyphen) and described
incus-compose: <name>. Its features are images, profiles and storage
volumes; networks stay in the default project. Measured on the VM:
| compose | Incus |
|---|---|
entry web-r0 | instance web-r0-1, an OCI application container |
labels: | user.label.<key>, plus user.label.incus-compose.project and .service on every instance |
environment: | environment.<KEY> instance config |
healthcheck: | user.healthcheck.{test,interval,timeout,retries,start_period}, run by ic-healthd, which writes user.healthcheck.status |
restart: "no" | boot.autostart=false, user.healthcheck.restart=no |
network default | managed bridge <project>-default in the default project, described incus-compose: <project>:default |
named volume html | custom volume vol-html in the project, prefetched from the image |
ports: "18080:80" | proxy device listen tcp:0.0.0.0:18080 connect tcp:127.0.0.1:80 |
ipv4_address on an external network | a static DHCP lease on that bridge |
incus-compose runs two sidecars by default. ic-dns answers service names;
its image is not public (ghcr answered 403 on 2026-10-01) and SHC has its
own resolver, so the executor passes --no-dns. ic-healthd runs health
checks, restart policies and service_healthy waits; by default one daemon
with an unrestricted Incus certificate serves the whole server from an
incus-compose project and bridge of its own, so the executor passes
--healthd-scope project, which gives each project its own
<project>-ic-healthd with a certificate restricted to that project.
2. The executor contract
Section titled “2. The executor contract”The node agent is the executor’s only caller (simplification §4.7). The
projection it hands over is one compose entry per replica, named
<service>-r<ordinal>. The render on next still defaults restart to
unless-stopped (modules/app/restart_default.go), so the executor stamps
restart: "no" on every entry itself (section 3).
Up reads the projection (compose.yaml, compose.yml, docker-compose.yaml or docker-compose.yml, docker’s order), translates it (section 3), writes the translation to a mode-0600 file in a private temp directory, and runs
incus-compose --file <tmp>/compose.yaml --project-directory <dir> \ --project-name <project> up --no-start --no-dns \ --healthd-scope project --network-driver bridgeso relative bind paths and .env still resolve against the projection’s
directory and the projection itself is never rewritten. Each entry carries
a spec digest label (shc.spec-digest) over its service, the networks,
volumes, configs and secrets, and the contents of the files it reads
(env_file, file-backed configs and secrets, the projection’s .env):
incus-compose never
recreates an instance whose definition changed. Before incus-compose runs,
Up deletes the instances of entries the projection no longer declares and
of entries whose digest moved, so nothing they hold (a published port, an
address) blocks their replacements (measured: a renamed entry took its
predecessor’s port 4001); afterwards every declared entry must have an
instance. Before deleting anything it refuses, by name, every key it
cannot honour (X400776INCKEY), every external network Incus does not have,
and a deployment network whose effective config (the ipam address and
every x-incus key) differs from the bridge incus-compose already made for
it (X400777INCCMP): incus-compose never changes an existing network.
Before deleting anything Up also makes sure every registry the images name
has a remote incus-compose can pull through (SHC-1356). incus-compose looks
an OCI image up by the Incus client remote named for its registry host
(lscr.io, ghcr.io, registry.example:5000); a remote it does not find
fails the pull (“no remote named lscr.io”), and a fresh node’s client
configuration has none but local and images. incus-compose has a few
built in (docker.io, ghcr.io, quay.io and three more); lscr.io is not one,
and the list is theirs to change. So the executor keeps the client
configuration incus-compose reads, its own, <state home>/incus-client
(/var/lib/shc/incus-client on a node, writable under the unit’s
ProtectSystem=strict; Options.ClientDir overrides it), and hands it to
every incus-compose run as INCUS_CONF, whatever the daemon’s own
environment says. Up adds a public OCI remote https://<host> there for
every registry the document’s images name, starting from Incus’s defaults.
An image is read as incus-compose reads it: <remote>:<image> when the
part before the colon names a remote it has (local:own), else an OCI
reference whose domain is the registry (docker.io for a bare name, none
for localhost/). A new registry is first asked GET https://<host>/v2/
(docker.io’s API at registry-1.docker.io, since docker.io itself redirects
to a web page): the distribution API answers 200, or 401 with how to log
in. Anything else, a redirect, a timeout (15 s) or a name that does not
resolve, fails the Up with X503789INCREG naming the registry, before any
instance is deleted or made, and nothing is written. A second Up of the
same images probes nothing and writes nothing, and the file is on disk, so
a reboot does not make the next Up ask again.
The operator’s place for a remote of their own (a registry mirror, a
private registry with its credentials helper) is the node’s global client
configuration, /etc/incus/config.yml (INCUS_GLOBAL_CONF), which
incus-compose merges under the executor’s file: a remote named there is
neither asked nor written over. incus-compose merges it only when its own
file exists, so the first Up writes that file even for a document whose
images all come from global remotes. A global file incus-compose would not
load makes it silently read Incus’s defaults alone, and the pull would fail
after Up had changed the node, so Up loads it as incus-compose does
(exactly one YAML document, default-remote a string, every remote’s
fields of their types and an address each) and refuses it with
X503789INCREG before anything changes. The shcd user’s own client
configuration (~/.config/incus) is not read for Up.
incus-compose makes everything and starts nothing (--no-start); Up then
starts what is stopped through the API: the project’s ic-healthd first,
then each entry after what it depends on, waiting for a service_healthy
dependency to read healthy (SHC-1290). incus-compose’s own start fails the
whole up when an instance is not running the moment its start returns,
so an entry that ran and exited at once (a crash, command: true) failed
the up where docker’s succeeds (measured on the lab). An application
container also reads Stopped for about 150 ms after its start returns, so
each start waits up to 3 s to see it running; one never seen running ran
and exited.
How long Up waits for health (SHC-1310): as long as the entry’s own
healthcheck allows, counted from the entry’s start, never a fixed bound.
Docker’s up -d waits for nothing but a service_healthy dependency, and
neither does Up: an entry nothing waits on is started and left to its
healthcheck. The window a refusal names is the healthcheck’s start_period,
then retries failed checks each interval apart and timeout long
(start_period + retries x (interval + timeout)), read from the keys
incus-compose writes on the instance (the compose file’s healthcheck or else
the image’s) with ic-healthd’s defaults, which are docker’s (interval 30 s,
timeout 30 s, retries 3, start_interval 5 s), for any it left out. Up waits
longer than that window, because ic-healthd jitters each due time by up to a
quarter of the interval: retries failed checks each up to 1.25 intervals
apart and timeout long, and, when there is a start_period, that period plus
one start_interval check straddling its end, itself up to timeout long. A
service_healthy dependency is waited on for that long, until it reads
healthy; it fails the dependent with X504794INCHLT, naming it and its
window, when it reads unhealthy, when it still reads starting past that
wait, or at once when it runs no healthcheck (as compose refuses one). An
entry with a lifecycle hook is started by incus-compose, which counts
--timeout from before the entry’s pre_start hooks and does not bound
those hooks, so Up gives that start 24 h, out of their way, and bounds the
health itself from the moment Incus first reads the instance running. One
that still does not read healthy when that wait has passed fails with
X504794INCHLT naming it and its window, and incus-compose’s start of it is
stopped. A post_start that fails on its own keeps incus-compose’s failure
(X500771EXECMD). Measured before this: ciso-assistant’s backend
(start_period 60 s, interval 30 s, timeout 10 s, 5 retries: a 260 s window)
is healthy 215 to 228 s after its first boot, past the two minutes.
Down, Ps, Exec and Events use the Incus REST API on the root-only unix
socket /var/lib/incus/unix.socket, never incus-compose, because
incus-compose refuses down without the compose file and a project that
moved off the node has none.
- Down finds the Incus project by its description, stops each instance with
the grace given (docker’s 10 s when none), forces the stop when the grace
runs out, deletes it (the project’s ic-healthd last), and deletes the
bridges described
incus-compose: <project>:. External networks are never touched. With volumes it deletes the project’s custom volumes in every pool, its images and the Incus project. - Ps lists the project’s instances that carry its label. The container ID is
<incus project>/<instance>. State maps Running, Stopped, Frozen and Error onto running, exited, paused and dead. Health is ic-healthd’s word for an instance that declares a test; none yet reads as starting. Incus records no exit code and no finish time for an application container, so a stopped one carries what the exec helper recorded when its first process ended (section 4, SHC-1300), andExitUnknown(-1) and no time where it left no record. The labels are theuser.label.*keys with the compose project and service under thecom.docker.compose.*names every other runtime uses. - Exec posts to
/1.0/instances/<name>/execwithwait-for-websocketand wires stdin, stdout and stderr through the operation’s websockets. It runs/.shc/exec-env --stdin <n> -- <cmd...>, the exec helper (section 4, SHC-1254). The environment never rides the request (SHC-1176): incusd merges the instance’senvironment.*config into an exec’senvironmentand copies the result onto the argv of its ownincusd forkexechelper, which any host user can read in/proc/<pid>/cmdlinefor as long as the exec runs (measured, section 8). So Exec blanks everyenvironment.*key in the request, an operator’s own included, and sends the entry’s staged exec environment (read from the node’s RAM) and then the caller’s variables as the first n bytes of the exec’s stdin, as NUL-terminated records. The helper reads exactly n bytes, builds docker’s environment from them and execve’s the command with it, so the command reads the caller’s stdin from where the records end; a stdin that ends inside them exits 125 and runs nothing. No shell reads a value, a name or value without a NUL arrives as given (discovery.type,OPTIND,PS4, a PEM), and the image needs no/bin/sh. The values never become a file inside the instance: the files API writes a stopped instance’s rootfs, the node’s disk. A caller’s bare name, an empty name and a NUL are refused (X400772EXEENV). The exit code is the command’s own, since the command replaces the helper; the helper’s own failures carry docker’s codes, 125 (it could not run), 126 (the command cannot be executed) and 127 (not found). A stdin that fails other than at EOF fails the exec, and so does an output stream that breaks other than at EOF, which is how incusd ends one (measured). The Incus API takes numeric ids only, so a user or group name is read from the running instance’s own/etc/passwdand/etc/groupthrough the files API, a user named without a group taking its primary group, as docker exec resolves them (SHC-2067); a name those files do not hold is refused (X400778INCEXE). No user means the image’s own (oci.uid,oci.gid), as docker exec does. An instance that binds no helper is refused, and an up replaces it. - Events subscribes to
/1.0/events?type=lifecycle&all-projects=trueand returns only once the subscription is live, so a caller that subscribes and then reads Ps misses no death (incus has no--since). incusd answers the websocket upgrade before it registers the listener, and registers it before its first keepalive ping, so Events waits for that ping or the first event (10 s at most). An instance that cannot be read for any reason but being gone closes the stream, so the caller’s reconnect and Ps recover the event instead of losing it.instance-shutdownandinstance-stoppedare deaths;instance-updatedis re-read and reported as a health event only when ic-healthd’s word changed. Instances without the project label (the ic-healthd sidecar) are ignored.
2.1 Networks and volumes the node makes (executor.Host)
Section titled “2.1 Networks and volumes the node makes (executor.Host)”A node runs replica projects (docs/proposals/2026-10-02-replica-projects.md),
whose networks and named volumes are all external: the node makes them
before the project that names them and removes them once none does. The
incus executor is an executor.Host (SHC-1290); hosttest.Run passes
against Incus 7.5.1 on shc-lab-1227 with NoExitStatus.
- Networks are managed bridges in the
defaultproject, which every incus-compose project uses for networks: the subnet’s gateway asipv4.address, the subnet’s first quarter left to static addresses (ipv4.dhcp.ranges). A network that states no IPv4 subnet, as the sharedmaster_default_traefik_publicdoes, takes the first /25 of the docker address pool (172.16.0.0/12) that no Incus network holds and the node neither has on an interface nor routes to, where docker puts the same network; left to Incus it got a random 10.x.y.0/24 inside the cluster’s address block (SHC-2229). NAT out unless internal (thenipv4.natandipv4.routingfalse),ipv6.address: noneunless asked, the MTU option asbridge.mtu, labels asuser.label.*, describedshc: <name>so Down, which deletes only what incus-compose describes as its project’s, never takes one. Incus names a bridge after its interface, 15 characters at most, so a longer name lives undershcplus 12 hex digits of its digest and answers to its own name throughuser.shc.name; Up hands incus-compose the Incus name. Another driver, an IPAM driver or options, any other option, an IP range or an aux address is refused. Incus refuses to delete a network an instance is on. - Volumes are filesystem custom volumes in the
defaultproject on the default profile’s pool, labelled asuser.label.*. incus-compose keeps named volumes inside each compose project’s Incus project, and Incus attaches no custom volume across projects, while a non-stateful pod’s volume is shared by its replica projects on a node. So Up turns every mount of an external volume into a bind of the volume’s host directory,/var/lib/incus/storage-pools/<pool>/custom/default_<name>, which incus-compose passes through as a disk device withshift=true(measured), and drops the declaration so incus-compose makes no volume of its own. That needs a pool whose volumes are directories for as long as they exist:dirorbtrfs. On zfs, lvm or ceph a volume is mounted only while attached, so CreateVolume refuses those pools. Incus does not count a mount by path as a use (measured: it deleted a volume a running instance had mounted), so RemoveVolume reads every instance’s expanded devices first and refuses while one mounts it. - Backup and quota. The volume’s data is a plain host directory with
the container’s own ownership (idmapped mount), which restic reads like
the incus-compose volumes above;
HostVolumePathnames it. No backup code reads a replica’s external volumes on any runtime yet (the backup path for replica projects is its own work). Incus enforces no size on adirpool’s custom volume and the Host sets none;incus storage volume inforeports its usage. - Labelled reads
/1.0/instances?recursion=1&all-projects=trueand filters onuser.label.*. Stop stops the project’s running instances, dependents first, with the timeout given or each one’s own grace, and forces the stop when it runs out; it removes nothing.
3. What the executor translates and refuses
Section titled “3. What the executor translates and refuses”incus-compose loads the compose spec but implements less of it than docker. Silently losing a key is the failure this section exists to stop.
| projection | incus-compose alone | the executor |
|---|---|---|
deploy.resources.limits.cpus | ignored | x-incus: limits.cpu.allowance: <cpus*100>ms/100ms |
deploy.resources.limits.memory | ignored | x-incus: limits.memory: <bytes>B (docker units are binary) |
deploy.resources.limits.pids | ignored | x-incus: limits.processes |
deploy.replicas other than 1 | scales | refused: compose never scales, one entry per replica |
any other deploy.* key | ignored | refused by name |
network driver_opts MTU | ignored | x-incus: bridge.mtu |
any other driver_opts key | ignored | refused by name |
network driver other than bridge | refused late | refused by name |
ipam subnet with no gateway | refused (“ipam gateway cannot be empty”) | gateway = the subnet’s first address |
| a network with no IPv6 asked for | Incus adds a ULA with NAT | x-incus: ipv6.address: none |
restart | ic-healthd enforces it, and boot.autostart follows it | stamped no: restarts belong to the node agent |
deploy.mode, endpoint_mode, restart_policy, update_config, rollback_config | ignored | dropped: placement, the resolver, the node agent and its rollout read them (as apple does) |
ulimits | dropped | x-incus: limits.kernel.<name>: <soft>:<hard> |
stop_grace_period | dropped | label shc.stop-grace, honoured by Down and by orphan removal |
any service key outside what incus-compose implements (dns_opt, read_only, …), an attachment key other than ipv4_address, ipv6_address and aliases, a network key, ipam key or top-level key it does not read | ignored | refused by name: the keys are an allowlist, so a new compose key is refused until it is handled |
network aliases | written as CNAMEs into the bridge’s raw.dnsmasq, left behind when an instance goes through the API, so a renamed entry fails dnsmasq as a duplicate (measured) | dropped: the node resolver publishes a deployment’s names |
extends, links, external_links | ignored | refused by name |
devices, cap_drop | dropped (measured) | refused by name |
extra_hosts | dropped (measured) | removed from the compose document incus-compose reads and carried in the entry’s staged file: the exec helper writes each entry into the instance’s /etc/hosts before its first process. host-gateway is the instance’s default gateway. Refused beside a hostname, whose hosts file is bound read-only, and when the container user is not root, which cannot write that file before the first process |
hostname | dropped (measured) | translated to raw.lxc lxc.uts.name = <name>, with lxc.mount.entry lines binding /etc/hosts and /etc/hostname read-only over Incus’s own files. The executor stages them at /run/shc-incus-hosts/<project>/<entry>/{hosts,hostname} by default, beside the runtime directory, outside the private environment tree: 0755 directories and 0644 files, holding no secrets. /etc/hostname holds the name and /etc/hosts keeps Incus’s localhost and IPv6 lines with 127.0.1.1 naming it. Start restores missing files from the unchanged projection after a reboot empties RAM; the entry’s files go when it leaves or the project goes down. Measured on home06, Incus 7.5.1, 2026-10-03 at 21:25Z: lxc.uts.name alone left m1 in both files and getent hosts buildkitsandbox found nothing; raw.lxc binds from a traversable directory put buildkitsandbox in both and resolved it to 127.0.1.1. Requires a string of 1 to 63 ASCII characters, with dot-separated alphanumeric labels and inner hyphens only; invalid values, unsafe staging paths and a hostname beside an authored x-incus raw.lxc lxc.uts.name are refused by name (SHC-1489) |
stop_signal | dropped (measured) | label shc.stop-signal: Stop and Down send it to the entry’s first process through the helper that supervises it (section 4); SIGKILL is a forced stop; a name that is no signal is refused, and so is SIGURG, the one signal the helper cannot pass on (its own runtime uses it, as docker’s signal proxy leaves it out) |
security_opt apparmor:unconfined / apparmor=unconfined | dropped (measured) | x-incus: raw.lxc: lxc.apparmor.profile = unconfined, LXC’s own unconfined in place of the profile Incus generates per instance (SHC-1221) |
security_opt seccomp=unconfined / seccomp:unconfined | dropped | x-incus: security.syscalls.deny_default: "false": with Incus’s default deny list off and no other syscall key, Incus loads no seccomp policy, as docker loads none. Beside an authored x-incus security.syscalls.* key or raw.seccomp it is refused by name: any such key keeps Incus loading a policy. A profile can set one too, so after incus-compose up --no-start Up reads the instance’s expanded config and refuses, before anything starts, one that would still load a policy; the entry carries the label shc.seccomp=unconfined, and Start checks the same before every start |
security_opt no-new-privileges (:true, =true; :false and =false are the default) | dropped | x-incus: raw.lxc: lxc.no_new_privs = 1, set on the first process and inherited by every process in the container. The list is read in order and the last no-new-privileges wins, as docker reads it, so a later =false clears an earlier =true |
any other security_opt (a named AppArmor profile, a seccomp file, label, systempaths, …) | dropped | refused by name: no faithful Incus form |
environment, env_file | environment.<KEY> instance config | resolved by the executor and staged in RAM, x-incus: raw.lxc: lxc.include = <file> (section 4); a value with a line break or NUL, a name with no value, a value that interpolates a variable, an authored raw.lxc, an x-incus environment.* key, a YAML anchor carrying an environment elsewhere (or an environment value aliased as one of its names) and a pre_start/post_start environment refused by name; every exec runs through the exec helper with docker’s environment (SHC-1254), and a pre_start image other than the entry’s is refused |
a projection’s .env | interpolated into instance config | refused: the render resolves every value |
top-level and service x- keys other than x-incus, x-incus-compose | kept in the compose incus-compose reads (an anchor there may hold an aliased environment) | dropped before anything is written |
service secrets | value pushed into the stopped instance’s rootfs | a read-only disk device on the file the render staged in RAM, plus /run/.shc-run-keeper when one binds under /run (section 4); uid, gid, mode, a source that is not an absolute file, a file off RAM, a target that is not clean or that Incus mounts its own file on, and two secrets at one target refused by name |
network_mode, pid, ipc | no shared namespaces | refused by name |
init: true, init: false | dropped (measured) | accepted and removed from the compose document: process 1 is Incus’s own init and the entrypoint is its child (measured, section 4), already providing docker’s --init behaviour. A non-boolean value is refused by name (SHC-1489) |
tty: true | dropped (measured) | accepted and removed from the compose document, and so is tty: false: a plain OCI instance’s entrypoint already has the console pty (/dev/pts/0) as stdin and stdout (measured on Incus 7.5.1, 2026-10-03), and the helper that supervises the entry starts the child with those same descriptors, its own stdin, stdout and stderr. A non-boolean value is refused by name (SHC-1489) |
read_only: true | no read-only root (measured) | refused: Incus 7.5.1 has no read-only root for a container: raw.lxc lxc.rootfs.options = ro fails the start (forklxc exit 1, measured 2026-10-03). read_only: false loses nothing and is not refused |
tmpfs without size= | sized at 32 KiB (measured) | refused |
depends_on with service_completed_successfully | dependent starts without waiting (measured) | the executor starts entries itself, waits for the dependency to end and takes only an exit of 0 from its record; one that failed or left no record fails the up, unless it is required: false (SHC-1300) |
entrypoint/command argument with a line break, or with both quote kinds | Incus accepts a line break and splits the command there (sh: -c requires an argument), and fails the start of both quote kinds: “Unable to encode command argument” (measured on 7.5.1) | the argument travels in the entry’s staged file and a token stands in its place; the helper puts it back (section 4, SHC-1229). One that would interpolate a variable is refused |
a one-shot job (shc.lifecycle=one-shot), restart: on-failure | Incus records no exit code, so a job that finished cannot be told from one that failed | admitted: the helper records the exit (SHC-1300) |
| an image whose entrypoint insists on being process 1 (s6-overlay v3, every linuxserver image) | exits at once: process 1 is Incus’s own init | the start is refused by name, X400786INCPID, saying it runs only on docker or podman (section 4) |
two ports publishing the same host port (tcp and udp) | one proxy device replaces the other (named by port) | refused |
a long-form publish (target, published, protocol, host_ip, mode: host, the only ports: entry a render emits) | forkproxy proxy device connecting from 127.0.0.1 | x-incus-compose: nat: true on the port: a NAT proxy device that DNATs to the instance and keeps the client’s address (SHC-1223) |
| a host port another entry of the projection, another Incus instance on the node (any project, any proxy device) or anything holding a socket there (docker’s or podman’s publish, a forkproxy, a daemon) already publishes, on the same protocol and an overlapping address | a NAT proxy binds no socket, so both instances start and the first DNAT rule takes every connection | refused before anything goes (X409781INCPRT, naming the holder); Up holds one node-wide lock from that check to incus-compose making the instances, and Start checks a stopped instance’s own ports the same way before it starts it. The probe binds each address the claim covers; a socket a running instance of the projection’s own holds (its loopback forkproxy, or a forkproxy from before NAT) leaves out only the address it holds, so a daemon on any other address of the node still refuses |
| a publish on every address, reached from the node itself at 127.0.0.1 | Incus’s NAT output rules leave the loopback out, so the node cannot reach it there, where docker’s publish answers | a forkproxy device of the executor’s, shc-loopback-<port>, listening on 127.0.0.1 beside the NAT device (a client there is local whichever way it arrives) |
a publish on a loopback host_ip | NAT cannot carry it | that forkproxy alone, out of ports: a forkproxy port ahead of a NAT port would set the connect address incus-compose shares across an entry’s ports to 127.0.0.1 |
an authored x-incus-compose proxy device, or any authored device named proxy-<n> or shc-loopback-<n> | incus-compose adds authored devices after its own, by name, so proxy-<published> replaces the NAT publish with a forkproxy | refused by name: a publish is a port |
| a proxy device a profile (or any Incus config) gives an instance | applied to every instance that selects the profile; a NAT one binds nothing, so two can publish one port | refused by name after incus-compose up --no-start, with the node’s publish lock still held, before anything starts: an instance may carry only the proxy devices its entry’s publishes make |
a short-form publish, a port range, sctp, an IPv6 host_ip, a mode other than host | published without the client address, or not at all | refused by name |
A value the projection already sets under x-incus wins over the
translation. The refusal is a placement fact as well as an Up error: the
engine is an executor.Refuser, so executor.Refusals feeds placement’s
ReplicaDemand.Refused (SHC-1200’s mechanism) and a replica whose compose
incus refuses never lands on an incus node, with the node’s reason naming
each key (tests/_ce/core/placement/refused_incus_test.go). Of the 107
catalog compose files, 56 services in 33 apps carry a command or
entrypoint argument Incus cannot encode (kubo’s ipfs-cluster, mailcow,
docuseal, hockeypuck, hyperswitch, fleet, immich, matomo, ollama,
gitlab-runner, hookshot and mailbox-reconciler among them). They were
refused by name until SHC-1229; each is admitted now and lands on an incus
node (tests/_ce/core/placement/refused_incus_command_catalog_test.go).
4. Wiring
Section titled “4. Wiring”Fabric. The render attaches every service to the external network
__shc_fabric (name shc-fabric) with a pinned ipv4_address. On an
Incus node that network is an Incus managed bridge the node’s fabric
reconcile creates, the Incus arm of today’s docker and podman bridge
create: ipv4.address=<the node's /24 gateway>, ipv4.nat=false,
ipv6.address=none, bridge.mtu=<fabric MTU>, and ipv4.dhcp.ranges
kept out of the pinned range. Incus turns each ipv4_address into a
static lease, so the pin holds across restarts. The executor only checks
the bridge exists. Docker’s gw_priority (the fabric bridge as the
default route, simplification policy slice 2) has no incus-compose
mapping; until it does, the executor must set the fabric NIC as the
instance’s only default route or reachability fails closed (open item).
Deployment networks. A projection’s own networks become
<project>-<net> bridges. The address plan (ruled 2026-10-01) gives
node-local deployment networks 10.64.0.0/10; the render owns the subnet
through ipam, and the executor hands it to Incus with a gateway and no
IPv6. Without ipam, Incus picks a random 10.x /24, which must not happen
on a fabric node.
Volumes. A named volume is a custom Incus volume. On the dir pool it
sits at /var/lib/incus/storage-pools/<pool>/custom/<project>_vol-<name>,
a plain host directory restic can read (measured: probe_vol-html held
the nginx html with the image’s own uid and gid). On zfs or btrfs pools the
path is a dataset mountpoint. UID shifting is idmapped mounts, so host-side
ownership is the container’s own. Backups capture that path; Incus
snapshots stay unused (section 6).
Env and secrets (SHC-1224). incus-compose writes environment: as
environment.<KEY> instance config. Incus keeps that in its database, in
every export, backup and snapshot, echoes it in every exec operation’s
metadata, and copies it onto the argv of every exec’s incusd forkexec
helper, ic-healthd’s health checks included (measured on the proof VM). It
pushes a compose secret into the stopped instance’s rootfs, which is the
node’s disk and goes into every export. So the executor hands incus-compose
neither, and no value is ever Incus config:
- The environment. The executor resolves each entry’s environment itself
(
env_fileentries in order,environmentover them, dotenv or raw,$$a literal$). A dotenv file is read by compose-go’s own rules (dotenv/parser.go, v2.16.1, the parser incus-compose links), so a value is the one docker compose hands the container:"ab\qcd"keeps its backslash, a quoted value may span lines,\nand the other escapes compose knows expand in double quotes only. A bare name (compose would inherit it from its own environment) and a$NAMEor${...}outside single quotes (compose would interpolate it) are refused, and no refusal quotes a value. The executor writes the result aslxc.environment = "NAME=value"lines into<runtime dir>-secrets/.incus/<project>/<entry>.conf, mode 0600 in 0700 directories, refused unless the directory is on tmpfs or ramfs (X500779INCENV). The entry’sx-incussetsraw.lxctolxc.log.level = warnthenlxc.include = <that file>, so Incus holds only the path. The log level line comes first because incusd in debug mode sets liblxc to trace, which logs every value it sets into the instance’slxc.logon disk (measured: three hits without the line, none with it). liblxc reads the file at every start, stop and exec, never writes its lines into thelxc.confit saves (an include’s lines stay out of the unexpanded config), and applies them in order with overwrite: the include is the last config Incus loads, so the first process gets the compose values over the image’sENV(with Incus’s ownTERM, the spec’sHOME,container, and noHOSTNAME: first-process parity is a follow-up). An exec gets none of that reliably: incusd fills an exec’s request with the image’sENVand its ownPATH,LANGand, for root,HOMEandUSER, which liblxc applies over the include. So every exec runs through the exec helper (next item). Up, Down and Start hold one lock per project for their whole run, in every engine of the process, and Start reads its instance again once it has the lock: a start that landed between Up’s read of the instances and its staging would run an instance on the environment Up was replacing, and no later Up would see it. An instance made from another projection directory is replaced, since Start restages from the directory it carries. The compose the executor hands incus-compose is written in the project’s RAM directory and holds no value:x-keys other thanx-incusandx-incus-composeare dropped (an anchor there may hold an aliased environment), a YAML anchor that puts a service environment’s value or name anywhere else (a label value or key, a command, anenv_fileentry, a config’s content, insidex-incus-compose) is refused by its path before anything else reads the values (each value scalar is swapped for a token, at any depth, and yaml.v3 decodes the document with its own alias and merge rules, so the tokens land where the values would), and no refusal quotes a value (a compose that does not parse is refused with its line only), and a projection’s.envis refused, since incus-compose interpolates it into instance config. Refused by name: a value with a line break or NUL (an LXC line cannot carry one, and Incus’s ownenvironment.*already mangles a line break when liblxc reads back the config it saved), a name with no value (liblxc and compose would copy a daemon’s environment in), a value that interpolates a variable, an authoredraw.lxc, anx-incusenvironment.*key (Incus config by another name), and a lifecycle hook’senvironment. wordpress’s multi-lineWORDPRESS_CONFIG_EXTRAis the one catalog case; it stays off incus until it is one line. The entrypoint is untouched. - Execs (SHC-1254). A runtime may refuse loudly, never run differently
from docker silently, so every exec in an instance, Exec’s, a health
check’s and a lifecycle hook’s, runs through the exec helper: a static
Go binary (
cmd/shc-exec-envin shc, cecore/executor/incus/execenv) the shc package ships for amd64 and arm64 at/usr/lib/shc/bin/exec-env, bound read-only from the node’s disk into every instance at/.shc/exec-env(disk deviceshc-exec-env). It holds no value. Each entry gets a second staged file,<entry>.env, bound read-only at/.shc/environment(deviceshc-environment): NUL-terminatedNAME=valuerecords, the image’sENVand then the compose environment. Up writes it afterincus-compose up --no-starthas made the instance and before anything starts, from the instance’s OCI runtime spec (<incus dir>/containers/<project>_<instance>/config.json, the only copy of the image’s environment the node keeps). It is owned by the node’s id for the entry’s exec user (oci.uidandoci.gidthrough the instance’svolatile.idmap.next; the same ids in a privileged instance) and mode 0400, so inside the instance only that user and root read it (measured: uid 1001 refused, uid 1000 reads). A compose health check becomes["CMD", "/.shc/exec-env", "--", <argv>]; aCMD-SHELLor a string test becomes/bin/sh -c <test>behind the helper, the image’s own shell as docker runs it; the image’s own check, which incus-compose writes at create, is rewritten the same way on the stopped instance. Apost_startcommand gets the helper in front; one written as a string goes through compose’s word split and is exec’d as argv, as docker compose does, never by a shell. Apre_starthook is a whole container spec that inherits every attribute of the service it does not declare,entrypointandcommandincluded (compose-go’s loader), and incus-compose execs its entrypoint followed by its command in an instance cloned from the entry’s config and devices; so the helper goes in front of the entrypoint the hook ends up with (its own, the service’s, or the helper alone) and the command is left as it is. That instance carries the entry’s labels but is not the entry: it never stages or re-owns the entry’s file, Ps does not list it and Start refuses it. The helper looks a command up as runc does (Go’sexec.LookPath): an emptyPATHsearches nothing, an empty element is the working directory, and a candidate counts only when access(2) gives the exec user execute permission; a failure names the command as given, never the pathPATHresolved it to. The helper builds docker’s environment and execve’s the command with exactly that: the image’sENV, the compose environment over it, an Exec caller’s over both, docker’s defaultPATHand theHOSTNAMEwhere none sets them, andHOMEfrom the exec user’s passwd entry (/without one) where none sets it, as runc does. Nothing incusd or liblxc adds (LANG,USER,TERM,container) and noenvironment.*an operator set in Incus reaches the command. The command replaces the helper, so its exit code, signals and ic-healthd’s timeout act on the command itself. Two cases cannot be told apart on the node, since the spec is umoci’s conversion of the image’s config and the config is not kept: umoci addsTERM=xterm, and aHOMEfrom the image user’s passwd entry when the image sets none. So a specTERM=xtermis dropped (docker gives an exec noTERMwithout a terminal; an image that itself setsTERM=xtermloses it in its execs), and a specHOMEequal to the image user’s passwd home gives way to the exec user’s (an image that setsHOMEto exactly that, exec’d as another user, gets the exec user’s). Every other case is docker’s. Up recreates a running instance whose exec environment would change, and Start writes a missing one again after a reboot. Before incus-compose runs, Up puts a placeholder where each entry’s device points (incus-compose needs the file to make an instance), unless a running instance it keeps reads the file; the helper refuses the placeholder with 125. An Up that stops between making or keeping an instance and preparing it leaves that placeholder, never an earlier environment, and Start then prepares the instance from its projection and rewrites the image’s own check before it starts it, while Exec refuses until one has. Refused by name: apre_startin another image than the entry’s (its instance binds the entry’s file, which holds the entry image’sENV), a hook as another user than the entry’s or root, a privilegedpre_startin an unprivileged entry that does not name root (its instance has no id map), apre_startbeside an isolated id map (its instance gets a range of its own, and Incus refuses an isolated map with a fixed base), authored or from the instance’s profiles, judged afterup --no-start, and a mount or device at/.shcor under the helper’s device names. So dify, ciso, ciso-assistant and greenbone, refused by name under SHC-1224 for aLANG,HOMEorUSERbeside a check, run on incus. - The first process runs under the same helper, which stays its parent
(SHC-1300). Incus records no exit code and no finish time for an
application container (measured on 7.5.1: its config,
/stateandlxc.loghold none, onlyvolatile.last_state.power), so afterup --no-startand before any start the executor puts/.shc/exec-env --supervise --in front of each entry’soci.entrypoint, the image’s or the compose file’s. Incus reads that key at every start (measured: a value patched on a stopped instance runs at the next start and after a restart) and splits it itself, so the helper’s words go in front of the text that is there. The helper builds docker’s environment from the entry’s staged file, as it does for an exec, starts the entrypoint as its child, passes every signal on, waits, writes<exit code> <finish time>to/.shc/status/exitthrough a rename, and exits with that code: the command’s own, 128 plus the signal for one a signal ended, 126 or 127 for one it could not start. So the first process gets docker’s environment too, where it used to get Incus’s own around the compose one (TERM, the spec’sHOME,container, noHOSTNAME).- The record’s home is a tmpfs of the entry’s own on the node (
size=4k, three inodes,nosuid,nodev,noexec), bound read-write at/.shc/statusand owned by the node’s ids for the entry’s user. It is the one place an entry’s processes write node RAM, which is why each entry gets a capped mount of its own instead of a directory in the shared staging tmpfs. Up mounts it, Start mounts it again after a reboot, and it goes with the entry or with Down. The executor clears it before every start it makes and the helper clears it again as it starts, so a start made past the executor leaves no earlier record either. It is read only for an instance that is not running, and only when the file was written no earlier than the instance’s last start, judged by the file’s own time on the node, not by the time the record claims. Linux stamps a file from its coarse clock, which can run one tick (1 to 10 ms, read from the kernel) behind the clock a start is read from, so a record up to one tick older than the start is still that run’s; Incus takes about half a second to handle an exit before it can start the instance again (measured), so no earlier run’s record is that close. An instance Incus never started (it reports the Unix epoch) reads no record at all: whether one in its directory is its own cannot be told, and that is an unknown exit. The directory is the tenant’s to write, on both sides: the executor opens the record without following a link or waiting on a pipe, takes a regular file only and reads no further than a record is long, and the helper removes whatever sits where it writes and makes each file anew, so a pipe, a link or a directory left there neither blocks it nor is written through. - No record is an unknown exit (
ExitUnknownand no time), never a clean one: a helper that was killed writes none. A stop that had to kill the entry (the grace ran out, or Incus’s own stop ran) took the helper with it, so the executor writes 137 itself, as docker reads a killed container, and a death event read before that write carries 137 too. - A stop signals the entrypoint from the node (section 2), which is the
helper, with the entry’s
stop_signal(labelshc.stop-signal, else SIGTERM); the helper passes it on and keeps waiting, so the process gets its own signal and its whole grace. - Ps, Labelled and Events read the record: a job (
shc.lifecycle=one-shot),restart: on-failureandservice_completed_successfullyare honoured instead of refused, andhosttest.Runpasses with exit codes on a live node. - Process 1. In an Incus application container process 1 is Incus’s own
init and the entrypoint is its child (measured), with or without the
helper; the helper adds one more parent and changes nothing about who
is process 1. A signal the app does not handle therefore takes its
default action, as under docker’s
--init. There is no opt-out key. An image whose entrypoint insists on being process 1 (s6-overlay v3, which every linuxserver image is built on) exits at once on incus. The executor watches a started entry for a moment, and one that ended with a failure and “can only run as pid 1” in its console log is refused by name: X400786INCPID, “insists on being process 1 … it runs only on docker or podman” (measured againstdocker.io/linuxserver/syslog-ng). Up’s starts and the node agent’s Start both watch, for a quarter of a second; an exit after that is a death with its exit code like any other. While Incus handles an exit it reads the instance running with no pid and refuses its console (“connection refused”, measured), so the executor waits for the instance to be down and reads the console again until it answers, all inside five seconds. An entry that failed at once and whose output cannot be read in that time fails the start as an engine failure that names it: what it said decides whether it can run here at all, so it is never taken for started. That is the text placement will show as a derived constraint; catalog apps known to need process 1 get an ordinary constraint keeping them off incus and apple once selectors land.
- The record’s home is a tmpfs of the entry’s own on the node (
- An argument Incus cannot carry in
oci.entrypointrides the staged file (SHC-1229). Incus accepts a line break and splits the command there, and fails the start of an argument with both quote kinds (measured). The executor takes each such argument out of the entry’sentrypointandcommand(and itspre_starthooks’) before incus-compose reads them, leaves a token (shc-arg-<position>-<digest>) in its place, and writes the argument as a record of its own in the entry’s staged file; the helper puts it back before it runs anything. An argument an author wrote in the shape of a token is carried the same way, so every token in an argv is one the executor made and each stands for exactly one argument. The one word the executor did not write is the image’s own entrypoint and command, which incus-compose puts inoci.entrypointbeside the entry’s: an instance whoseoci.entrypointholds a token more often than the entry’s own argv placed it is refused by name (X400776INCKEY) and removed, because the helper would put the argument there too. A string is split the way compose splits it first:executor.SplitWordsis compose’s own parser (mattn/go-shellwords 1.0.12, no expansion) kept rule for rule, held against it on three million random strings, so a string the executor splits is the argv every other runtime runs (an unquoted;ends the command there, a backslash in double quotes takes the next character as it is). The argument is read as compose would have read it ($$is a literal$, and one that names a variable is refused), the token is made from the argument’s text, so the spec digest follows it, and the staged file is what Up already compares to decide a recreate. - The digest is a label any reader of the instance sees, so it carries the
environment’s names and never its values, and no secret’s content. Up
recreates a running instance whose staged file differs or is gone (it
may hold any environment), rewrites the file of a stopped one, and drops
the files of entries that leave; Down drops the project’s. A removal
puts an empty include back for the stop, since Incus loads
raw.lxcon every stop and a missing include fails it, and takes it away after. - A reboot empties RAM, and the node agent restarts replicas without an
Up. Every entry carries
shc.projection, its projection’s directory, so Start writes a missing file again from that projection, only when it still hashes to the instance’s spec digest (X503780INCRST otherwise; the next Up recreates it). - A service’s secret becomes a read-only disk device
shc-secret-<n>-<secret>on the file the render staged in RAM (the docker runtime’s SHC-1159 staging), at/run/secrets/<target>or an absolute target, as docker compose binds it; a rewrite of the staged file reaches the instance in place. Incus mounts a tmpfs on/run, after the devices, when the image’s/runis empty, and a bind at/run/secrets/<name>goes under it (measured on busybox). So an entry that binds a secret under/runalso gets the config/run/.shc-run-keeper, a fixed line of text incus-compose pushes into the stopped rootfs before every start:/runis never empty and stays the image’s own, as docker has it. A raw.lxc mount entry instead fails: liblxc mounts it inside the container’s user namespace, which cannot pass the 0700 staging directories (measured). A secret reads as owner 65534 and mode 0444 inside (the bind is not idmapped), where docker shows root. Refused by name: a secret withuid,gidormode(a bind keeps its file’s own), one whose source is not an absolute file, a file off RAM, a target that is not a clean absolute path below/, one at/etc/hosts,/etc/hostnameor/etc/resolv.conf(Incus mounts its own file there after the devices), two secrets at one target.
The Incus API listens only on the root-only unix socket, plus the one HTTPS listener ic-healthd needs (below), and no SHC component other than shcd holds an Incus certificate. The projection on disk still holds the rendered values until projections carry references (scheduler §3.24): an env-only secret is in the rendered compose the render persists, which the executor reads and never copies. Carrying references there is the render’s change and waits on the ruling about secrets in rendered specs.
Health. ic-healthd is used for health only: with restart: "no" it
writes user.healthcheck.status and never acts. ic-healthd’s own HTTP
listener (/metrics, /health, /ready) is bound to the sidecar’s loopback by
the executor (SHC-1228). It dials the Incus HTTPS API (core.https_address);
bind that to the address the project bridges can reach and let the node’s
nft policy admit only the bridges. The node agent reads status through Ps
and Events, records it per replica, and owns every restart.
Resolver. --no-dns keeps ic-dns off. The render’s dns and
dns_search go through incus-compose unchanged (it implements both) and
point instances at the node resolver.
5. Node facts and conditions
Section titled “5. Node facts and conditions”An Incus node reports runtime: incus and the facts docker nodes report.
Two are Incus-specific:
- Foreign instances. Instances SHC did not create (no Incus project
described
incus-compose: <an SHC deployment>) still use the host. The node agent counts them and theirlimits.cpu*andlimits.memory(or, unlimited, a configured share) and subtracts that from allocatable, so the fit gate (scheduler §3.3) never packs onto memory Incus already gave away. - Hypervisor not quorate. When the node is a VM on a Proxmox cluster
that lost quorum (the node provider’s view), or the Incus API stops
answering, the agent sets the node condition
hypervisor-not-quorate. New replicas are not placed there; running ones stay, because the agent, not the hypervisor, decides. It clears when quorum and the API return.
6. Clash register
Section titled “6. Clash register”Every place Incus (or incus-compose) and SHC both want to decide, and who wins.
| area | Incus wants | SHC wants | winner and mechanism |
|---|---|---|---|
| restart policy | boot.autorestart, ic-healthd restarts on failure | node agent restarts (§3.8) | SHC. The executor stamps restart: "no" whatever the render wrote; ic-healthd health-only (measured: user.healthcheck.restart=no, a killed daemon stays down) |
| boot start | boot.autostart per instance | a rebooted node starts nothing until the agent says so (SHC-1112) | SHC. restart: "no" gives boot.autostart=false |
| healing and moves | Incus cluster healing, evacuation | the scheduler moves replicas | SHC. Standalone Incus per node, never an Incus cluster; no evacuation |
| placement | Incus cluster placement groups | SHC placement engine | SHC. No Incus cluster, so Incus never places |
| DNS | ic-dns zones, bridge dnsmasq names | node resolver, TTL 5 s, nearest ready replica | SHC. --no-dns; render’s dns/dns_search |
| firewall | Incus network ACLs, bridge nft rules for NAT and DHCP | SHC’s nft policy compiler (SHC-1202) | SHC for policy. Incus keeps only its bridge plumbing; no Incus ACLs are written |
| addressing | random 10.x /24 and an IPv6 ULA per bridge | address plan: fabric 10.128/10 per tenant, deployment nets 10.64/10 | SHC. ipam subnet with gateway, ipv6.address: none, fabric bridge made by SHC |
| published ports | proxy devices (forkproxy, source becomes 127.0.0.1) | --port publishes on the node or through a load balancer, with the client’s address | SHC (SHC-1223). Every publish is a NAT proxy device (nat: true): Incus’s nftables rules DNAT fib daddr type local (or the host_ip) on the port to the instance’s static address, the fabric pin, or with none the address it finds on the bridged NIC, and masquerade only a connection from the instance to itself, so the client’s address arrives. The device belongs to the instance, so a recreate on a moved digest makes it again on the same pin. incus-compose skips a NAT device, with a warning only, when Incus lacks instance_port_forward (static address) or oci_network_config (none), and it picks the extension per port from one connect address the entry’s ports share: with no static address the first port sets it to 0.0.0.0, so every later port needs instance_port_forward too. Up reads the server’s extensions and refuses first, two or more unpinned publishes needing both. A NAT proxy binds no socket, so Up also refuses a host port already published on the node (X409781INCPRT): by another entry, another instance’s proxy device in any Incus project, or any socket (docker, podman, a forkproxy, a daemon), probed by binding it as docker would. The reply leaves by the instance’s default route; while that is not the fabric NIC (section 4), the reply to a connection DNATed to the fabric pin crosses another bridge, which a strict reverse-path rule would drop. Measured on a lab VM: the client address arrives before and after a recreate and with another default route; the node’s loopback is not published |
| storage pools | pool choice, x-incus-compose.pool | one local pool per node, restic captures | SHC. One pool named in node config; no LINSTOR in this ticket |
| snapshots | instance and volume snapshots, scheduled snapshots | one backup capture path (§3.7) | SHC. No snapshots.schedule; restic reads the volume path |
| images | an incus-compose-cache project that keeps every pulled image | image GC by the node agent | SHC (SHC-1222). image_prune_cron, node-local, grace 1h: an image in incus-compose-cache, and one in a deployment project the executor made (an instance there carries shc.spec-digest; an operator’s own incus-compose project is left alone), is removed once no instance on the node (running or stopped) names its fingerprint in volatile.base_image and the newer of uploaded_at and last_used_at is older than an hour. A prune and an Up exclude each other, so an image an Up has found is not removed before its instance exists. Not --image-cache "": that only moves the copies into each project, and every recreate would pull the image again |
| health daemon | one global ic-healthd with an unrestricted certificate | least privilege | SHC. --healthd-scope project |
| env and secrets | environment.* config, echoed in exec metadata, in exports and backups; an exec’s environment copied onto incusd’s forkexec argv; secrets pushed into the rootfs | secrets on tmpfs only, never on the node’s disk, an argv or Incus config | SHC (SHC-1224). The environment is staged in RAM and included through raw.lxc (lxc.include), Start restages it after a reboot, a secret is a read-only bind of its staged file with /run kept off a tmpfs, Exec’s own environment rides its stdin (section 2), API root-only. Measured on a lab VM with incusd in debug mode: no value in instance config, the instance query, an exec operation, an argv, the rootfs, an export, /var/lib/incus or /var/log/incus. Every exec, a health check’s and a hook’s included, runs through the exec helper with docker’s environment (SHC-1254) |
| exit codes | none recorded | the agent reads exit codes for restart policy | SHC (SHC-1300). The exec helper supervises each entry’s first process and records its exit code and finish time on a per-entry tmpfs; ExitUnknown only where it left no record |
7. What incus-compose must not be used for
Section titled “7. What incus-compose must not be used for”incus-compose down (needs the compose file), restart, stop and
pause (they write user.healthcheck.stopped and expect ic-healthd to
act), backup and its x-incus-compose.backup, --scale, and the
global healthd. The executor calls up and nothing else.
8. Proof
Section titled “8. Proof”The VM was VM 300 on pve03 (Debian 13, 6 GB, 4 cores, 30 GB, on a
host-only bridge as the SHC-1191 spike networked its guests), Incus 7.5.1
from Zabbly, incus-compose built from source, and a dir storage pool. A
fabric bridge shc-fabric stood in for the node’s fabric reconcile
(10.84.99.1/24, no NAT, no IPv6). The executor ran through a small Linux
driver over executor.New("incus"), not through incus-compose by hand.
A catalog app through the executor (2026-10-01 22:35Z, executor e2e1794c).
kubo’s compose rendered into a two-entry projection: kubo-r0 (healthcheck,
two named volumes, ports: 4001 mode: host, a fabric pin 10.84.99.10, an
ipam-named deployment network, cpu and memory limits, the render’s
restart: unless-stopped) and ipfs-cluster-r0 (its own volume,
depends_on kubo-r0 service_healthy, a fabric pin 10.84.99.11, its
multi-line command made one line).
up rc=0 in 32s; second up rc=0 in 2s (converged, nothing recreated)kubo-r0-1 boot.autostart=false healthcheck.restart=nodefault-kubo-proof/kubo-r0-1 state=running health=healthydefault-kubo-proof/ipfs-cluster-r0-1 state=running health=healthykubo-r0-1 10.84.99.10 (fabric) 10.64.20.232 (deployment network)/var/lib/incus/storage-pools/default/custom/default-kubo-proof_vol-ipfs-data: api autoconf blocks config datastoreproxy-4001: listen tcp:0.0.0.0:4001 connect tcp:127.0.0.1:4001; tcp 4001 openipfs-cluster's REST API answered on its fabric pin 10.84.99.11:9094exec with PROOF_SECRET: env=s3cr3t-value stdin=stdin-ok exit=4; host argv hits for the value 0; env files left in the instance 0kill the kubo daemon: kubo-r0-1 state=exited exit=-1, STOPPED 25 s later (nothing restarted it)events: {"Kind":"die","Container":{"ID":"default-kubo-proof/kubo-r0-1"},...,"ExitCode":-1}down with volumes rc=0: project, bridges and volumes gone, shc-fabric untouchedBefore the exec fix the same exec put the value on incusd forkexec ... -- env ... PROOF_SECRET=s3cr3t-value, mode 0444, readable by an unprivileged
host user (/proc has no hidepid).
A third round (23:20Z, executor a8568891) on the same projection:
exec into ipfs-cluster-r0-1: inherited=proof-cluster-secret-... (its compose environment) supplied=s3cr3t-value stdin=stdin-ok exit=4; host argv hits 0Events: "subscribed" after Incus's first keepalive; the kill was reported as a diethe deployment network moved to 10.64.21.0/24: X400777INCCMP "network public already exists as ic-wx3gwokevn with ipv4.address 10.64.20.1/24 -> 10.64.21.1/24", nothing touchedkubo-r0 renamed kubo-r1, same port 4001: up rc=0 in 19s, kubo-r1-1 RUNNING with proxy-4001down with volumes rc=0, nothing left Earlier runs on the same VM proved thedigest converge (an entry whose spec changed is recreated, an unchanged one left alone), the refusal of an authored multi-line command before anything ran, and the busy-instance retry on delete (ic-healthd writes status while an instance stops).
mailcow through incus-compose. shc/apps mailcow’s compose, rendered with
its vars (18 services; mailcow-apppass left out because its image lives in
the corp registry the VM cannot reach), went through incus-compose by hand:
all 17 instances were created in 116 s. Per key:
| key | result |
|---|---|
cap_add (dovecot, postfix NET_BIND_SERVICE) | pass, trivially: nothing is written, and root in an unprivileged Incus container already has the full set (CapEff 000001ffffffffff) |
sysctls (redis net.core.somaxconn=4096) | pass: linux.sysctl.net.core.somaxconn: "4096", and 4096 inside redis |
ulimits (dovecot nproc 65535, nofile 20000:40000) | fail in incus-compose (dropped; the container sees nofile 1048576, nproc unlimited); pass through the executor’s limits.kernel.* (20000, 40000, 65535 measured) |
mode: host ports (dovecot 110/143/993/995/4190, postfix 25/465/587) | pass as proxy devices listen tcp:0.0.0.0:<port>; the client address arrives as 127.0.0.1, which mail cannot accept |
network aliases | accepted, written into the bridge’s dnsmasq; the executor drops them and the node resolver must publish them |
hostname (rspamd) | fail: dropped, the hostname is the instance name |
depends_on service_started / service_healthy | pass: the dependent waits for the healthcheck (“Waiting for the healthcheck”) |
depends_on service_completed_successfully | fail: the dependent starts at once and Incus records no exit code |
tty: true (dovecot) | fail: dropped (measured: the entrypoint’s stdin and stdout are /dev/pts/0, the console pty). The executor accepts it: the helper that supervises the entry starts the child with its own stdin, stdout and stderr |
stop_grace_period (mysql 45 s, rspamd 30 s) | fail: dropped; the executor carries it as a label |
command with a line break or both quote kinds (11 of the 17 services) | fail at start: “Unable to encode command argument” |
restart: unless-stopped (every long-running service) | ic-healthd would enforce it; the executor stamps no |
Through the executor, the same compose was refused before anything ran,
naming each of those commands, every service_completed_successfully,
dovecot’s tty and rspamd’s hostname (X400776INCKEY). The commands and
the completed dependencies are honoured since SHC-1229 and SHC-1300, and
the mail ports keep the client address since SHC-1223. rspamd’s hostname
is translated since SHC-1489. dovecot’s tty was dropped by incus-compose
(measured: the entrypoint’s stdin and stdout are /dev/pts/0, the console
pty). The executor accepts it: the helper that supervises the entry starts
the child with its own stdin, stdout and stderr, so the entry keeps that
terminal.
Through shcd (SHC-1225, 2026-10-02 03:20Z). A second Debian 13 VM with
Incus 7.5.1, incus-compose and no docker or podman ran the lane’s shcd and
shc. shc cluster init --runtime incus recorded the node’s runtime, and
shc install hello --runtime incus (nginx with a healthcheck) went through
the deploy funnel into the executor:
installed stack "hello"; shc ls: running; shc status: healthy 1/1, runtime incusincus: prod-default-hello/web-1 RUNNING, boot.autostart=false, user.healthcheck.restart=no, user.healthcheck.status=healthythe instance stopped behind shcd's back: shc ls degraded, nothing restarted it (this build predates the SHC-1112 node agent, which owns restarts)shc update (no runtime stated, the sidecar carries incus): web-1 RUNNING againa healthcheck change: spec digest 977d27a9 -> 136ffffd, instance recreatedshc uninstall: the Incus project gone, nothing leftcompose stated on the incus node: X400955RTMPINno runtime stated: X409179PLCPND, admission places an unstated deploy as composeThe same binary on a docker VM ran the same app through docker compose, and
--runtime incus there was refused with X400955RTMPIN. A two-node cluster of
the two VMs was not formed: an unlicensed estate refuses clustering
(X403029LICENT), so the mixed-cluster criterion rests on the unit tests until
the rig returns. The deb could not install cleanly on the incus VM: it
hard-depends on docker and its postinstall refuses a host with neither docker
nor podman (SHC-1239).
security_opt (SHC-1221). Measured first on a lab VM (Incus 7.5.1) with
stand-ins carrying the catalog’s sets: apparmor:unconfined (frappe’s bench
services) runs unconfined with Incus’s seccomp filter on (Seccomp: 2), as
docker keeps its default filter; seccomp=unconfined with
apparmor=unconfined (greenbone’s ospd-openvas) runs unconfined with no
filter (Seccomp: 0) and opens a raw ICMP socket; no-new-privileges:true
shows NoNewPrivs: 1 under Incus’s generated profile; a service with none
runs under Incus’s generated profile, enforced. Every container keeps the
full capability set (CapEff 000001ffffffffff), so cap_add changes
nothing. The 29 catalog services in 9 apps that carry security_opt (28
apparmor:unconfined, one apparmor=unconfined, one seccomp=unconfined)
are no longer refused
(tests/_ce/core/placement/refused_incus_catalog_test.go).
Then the real images on a rig cell’s incus node (Incus 7.5.1, 2026-10-07),
installed through shc install --where node=<the incus node>:
frappe: frappe-web, frappe-queue-long, frappe-queue-short, frappe-scheduler and frappe-socketio RUNNING and healthy; attr/current reads the instance's own incus-<project>_<instance> profile with :unconfined (enforce) inside it; Seccomp: 2; NoNewPrivs: 0greenbone: all 20 replicas ready or completed; ospd-openvas-r0-1 RUNNING, raw.lxc holds lxc.apparmor.profile = unconfined; attr/current reads unconfined; Seccomp: 0; NoNewPrivs: 0Not measured: whether frappe needs apparmor:unconfined at all under the
profile Incus generates. The translation stays the faithful one.
9. What is left
Section titled “9. What is left”- An unstated runtime: admission places a deploy that names no runtime as
compose, so on an incus-only cluster
shc installwithout--runtime incusstays pending. Whether an unstated runtime may land on any runtime whose executor admits the projection is not ruled. - The render: always name a deployment network’s subnet (
ipam), stamprestart: "no"(the executor does it today), and publish network aliases to the resolver. - The node agent: the foreign-instance fact and the
hypervisor-not-quoratecondition (section 5 is design only). Restarts are built: the agent starts the one instance (cemodules/nodeagent, Start on the incus executor). - The node’s fabric reconcile is built in another shape (SHC-1322, SHC-1227): each
tenant’s bridge is an unmanaged kernel bridge from the tenant-bridge
ledger (ce
modules/fabric/kernelbridge.go), and the fabric NIC is routed and the only default route (cecore/executor/incus/fabric.go), where this page planned oneshc-fabricIncus bridge per node. - Published ports measured on a lab VM (SHC-1223): a connection from
another host reaches the replica with that host’s address, before and
after the executor recreates it on a moved digest, and with a second
network as the instance’s default route (the node’s
rp_filteris 0; a strict reverse-path rule there is not measured). A connection to the node’s loopback is not DNATed (Incus’s output rule excludes 127.0.0.0/8), where docker’s userland proxy answers it; the node’s own address works. Still owed: mailcow’s postfix on a rig. - incus-compose always makes a
defaultbridge for the project (ic-healthd sits on it) and, with noipamfrom the render, Incus picks its subnet at random (measured: 10.223.193.0/24), outside the address plan; pin it throughx-incus-compose.healthdor the render. - ic-healthd’s HTTPS listener confined to the project bridges by the nft policy (SHC-1202).
- Exit records live in node RAM, so a reboot takes them with everything else there: a job that finished before the reboot reads an unknown exit after it, where docker would still read 0. The node agent decides what an unknown exit means for a one-shot.
- Private registries (SHC-1411, on the registry connections of SHC-795 and
SHC-800): a docker-registry connection for a registry the images name is
an OCI remote in the executor’s client configuration, non-public, whose
credentials_helperis the executor’s own helper (<client dir>/registry-auth, mode 0755). incus-compose runs it once per remote (<helper> get, the host on stdin) and caches the answer for the run. The helper holds no secret: it accepts a host of[A-Za-z0-9.-]+with an optional port and cats<auth dir>/<host>.json({"Username","Secret"}, mode 0600). That directory isOptions.AuthDir(/run/shc/incus-registry-auth); Up writes the file before incus-compose runs and removes it when the Up ends, and refuses X503789INCREG, writing nothing, when the directory is not a tmpfs or ramfs. Logins are read at Up from the same docker-registry connections the docker engine uses, keyed by the registry host (Docker Hub’s spellings fold todocker.io). A registry with no connection and no operator remote is checked per image, anonymously:/v2/answering 200 is public, and a Bearer challenge is followed for a token scopedrepository:<path>:pulland aHEADof the manifest. 401 or 403 refuses X503789INCREG naming the registry and the image, before anything changes; a timeout, a 404 or a 5xx does not, and a public verdict is remembered for the process so the next Up asks nothing. A remote the operator wrote in/etc/incus/config.ymlfor the same registry still wins and is left untouched.
10. Deltas owed to the simplification proposal
Section titled “10. Deltas owed to the simplification proposal”- §4.7: the incus executor talks to the Incus API for everything but Up;
incus-compose’s
downcannot run without the compose file. - §3.8:
restart: "no"is load-bearing on incus beyond docker: it is also what keeps ic-healthd andboot.autostartfrom acting. The render still defaultsrestarttounless-stopped, so the incus executor stamps it. - §4.7: an executor’s
Execenvironment cannot ride the Incus exec API, which copies it onto a host argv; the contract holds by sending it on the exec’s stdin to a static helper that execve’s the command with it, with no file and no shell. - §4.4 (render): a projection network without
ipamwould get a random Incus subnet anywhere in the address block, so the incus executor never leaves it to Incus: it takes a free /25 of the docker address pool (SHC-2229).