A tenant container runtime: what "this app spawns containers" actually needs
Status: built; this note is still its only description. The verb is
shc.container.runin cemodules/containerbroker. Written 2026-08-08 while takinggitlab-runneroff the host Docker socket, so that the capability gap that work exposed is recorded once, with its real cost, instead of being re-derived badly the next time an app package reaches for/var/run/docker.sock.
Read the status line before quoting anything below as a fact about SHC.
This document is the answer to the question
docs/proposals/2026-08-08-in-stack-exec-broker.md §5 explicitly declines to
answer. That proposal covers “run a declared verb in a named sibling service”.
This one covers the other shape: create containers that do not exist yet,
from images chosen at run time. They are different problems and neither
document should be cited for the other.
1. The gap, stated as narrowly as it actually is
Section titled “1. The gap, stated as narrowly as it actually is”An application’s core function is starting containers, from images it does not know until the work arrives, with configuration the platform did not author.
Two packages in the catalogue are this shape:
| package | what it wants the runtime for |
|---|---|
gitlab-runner | the docker executor: one container per CI job, image named in .gitlab-ci.yml |
windmill | windmill-worker docker-mode jobs: hands the engine to user-authored scripts |
Everything else that reached for the socket wanted something narrower and has
since been moved off it without any new capability: matrix (exec into a
distroless sibling → context: service + argv-direct), mailcow
(ofelia-mailcow read it and never ran; dockerapi-mailcow only pinged at
import).
What is emphatically NOT the gap: the socket itself. Measured on gitlab-runner, three ways:
executor = "shell": job ran toJob succeededwith no container runtime present at all, as uid 999, and could not read the 0600config.tomlcarrying the runner’s auth token.executor = "docker"+host = "tcp://…": the full docker executor against a sibling engine. Job reached its script and succeeded; the bind-watcher on the host daemon recorded zero job containers.hostis a first-class runner config key. The socket was never required to reach a daemon, only to reach this node’s daemon.
So the package could ship without the socket, and now does. What remains unavailable is narrower and worth stating exactly:
per-job container isolation, for a TENANT, on a SHARED SHC node.
2. Why the socket cannot be the answer, restated with numbers
Section titled “2. Why the socket cannot be the answer, restated with numbers”Not a principle: a measurement. One trivial CI job (GIT_STRATEGY=none)
routed through a logging unix-socket proxy produced 225 Docker Engine API
calls. Among them, on every job, before any user script runs:
GET /networks → then GET /networks/<id> for EVERY networkOn the box this was measured, that walked 60 networks: every other
stack’s network, inspected individually, twice per job. GET /networks/<id>
returns each attached container’s name and IP. There is no opt-out; it is how
the executor decides what to attach to.
The compiled client surface is the whole API: ContainerCreate/Start/Exec/ Kill/Remove, ImagePull/Load/Import/Tag, Network*, Volume*. ImageLoad
alone is “push an arbitrary tarball into the host image store”.
A socket bind is therefore not “docker access”. It is read access to every
tenant’s topology on the node, plus write access to every tenant’s
containers, which is host root by a shorter path than the usual -v /:/host
argument. :ro buys nothing: the API is request/response over the socket.
A docker-socket-proxy does not fix this either. The proxy container still
needs the raw bind, which the sandbox refuses for a tenant; and filtering HTTP
verbs on an API whose object namespace is the whole node still leaves the
object namespace the whole node.
3. Capability 1: runtime: nested
Section titled “3. Capability 1: runtime: nested”A deployment-level declaration: x-shc-runtime: nested in app.yaml, or a
package-level capabilities: [nested-runtime]. SHC, not the package
author, materialises the sidecar: picks the dind image, sets
privileged/security_opt/devices, puts it on an SHC-minted network
reachable only from that deployment, never publishes its port, and injects
DOCKER_HOST=tcp://<alias>:2375 into the consuming service.
The package never writes privileged: itself. That key stays refused
everywhere, which is the point: the privilege is granted by the platform, to
a container the platform authored, or not at all.
Blast radius, stated honestly
Section titled “Blast radius, stated honestly”On a stock kernel a privileged dind container is host root. Rootless dind was tested as the way out and is not one:
docker:28-dind-rootless, no --privileged, with --security-opt seccomp=unconfined --security-opt apparmor=unconfined --device /dev/fuse --device /dev/net/tun
daemon boots clean: API listen on [::]:2375, docker version → 28.5.2, uid=1000(rootless), Privileged=falsefirst container: runc create failed: error mounting "proc" to rootfs at "/proc": flags=MS_NOSUID|MS_NODEV|MS_NOEXEC: operation not permittedThree retries, same error. SHC nodes have no userns-remap (docker info
SecurityOptions: apparmor, seccomp=builtin, cgroupns), so rootless dind
cannot start a single container.
Therefore granting this to a tenant on a shared node is equivalent to giving them the socket, minus the SHC daemon’s own API. It is narrowly better in exactly one respect: the 60-network enumeration and the ability to stop neighbours’ containers go away, because the tenant reaches a different daemon.
That one respect is not enough on its own. The capability is only defensible combined with placement:
nested-runtimemust imply NODE EXCLUSIVITY: a deployment holding it is pinned to a node hosting no other tenant.
That is the real cost and it should be priced as such: this capability sells a node, not a feature.
Authorisation
Section titled “Authorisation”Not a var. A tenant-scoped entitlement flipped by a platform admin (the master
tenant), recorded in the audit log at grant time and again at every install
that consumes it. Node exclusivity enforced at placement, not documented.
doctor reds a node carrying a nested-runtime deployment alongside another
tenant’s stack.
Prerequisite for it being genuinely safe
Section titled “Prerequisite for it being genuinely safe”userns-remap on the SHC daemon, or a sysbox / gVisor runtime class. Neither
exists on SHC nodes today. Until one does, nested-runtime is honestly
described as “host root, confined to a node we agreed to burn.”
4. Capability 2: an ephemeral-container broker
Section titled “4. Capability 2: an ephemeral-container broker”The general answer, and the one worth building.
A daemon API (POST /api/method/shc.container.run) accepting (image, argv,
env, mounts from a declared set, network = this deployment’s network,
timeout) and returning a handle supporting attach / exec / wait / kill. It is
context: container (ce:modules/job/context_container.go, today an internal
hook executor that already synthesises a temp compose project and runs
docker compose run --rm) promoted to an authenticated, tenant-scoped,
quota-charged API. The daemon keeps sole possession of the engine endpoint;
the caller never sees a socket.
Why gitlab-runner specifically can consume it
Section titled “Why gitlab-runner specifically can consume it”The runner ships a custom executor, confirmed compiled into the pinned
image (gitlab-runner/executors/custom), with config keys config_exec /
prepare_exec / run_exec / cleanup_exec (+ *_timeout), the job
environment delivered as CUSTOM_ENV_<KEY>, and BUILD_FAILURE_EXIT_CODE /
SYSTEM_FAILURE_EXIT_CODE conventions.
Four small SHC-authored scripts in the runner image translate those four stages into four broker calls. No fork, no patch, an upstream-supported extension point.
Blast radius
Section titled “Blast radius”Bounded by what the broker refuses:
- image-pull policy per tenant;
- mounts only from
ResolveVolume-approved sources, so@output/../../..and/var/run/docker.sockare refused by the same one policy, not a second one that has to be kept in sync; - attaches only to the deployment’s own network, so
GET /networksis not expressible; - CPU / memory / count charged against the tenant quota; concurrency and lifetime hard-capped.
A compromised runner gets “spawn containers inside my own blast radius” (which is the actual requirement), not “administer the node”.
Authorisation
Section titled “Authorisation”A capability on the deployment, granted at install by a platform admin and carried on the deployment row. Checked on every call against the calling deployment’s identity as the daemon sees the peer, not a bearer token in a config file a job script could read. Every call audited with image + argv digest.
What it is not
Section titled “What it is not”Not a socket proxy. A proxy filters verbs on an API whose object namespace is the whole node; the broker’s object namespace is the deployment. That difference is the entire security argument.
5. Honest open questions
Section titled “5. Honest open questions”- Swarm. The broker has to place the ephemeral container somewhere. On a
swarm estate “the deployment’s own network” is an overlay and the placement
node is not the caller’s node. Unresolved, and it is the same unresolved
question
in-stack-exec-broker.md§4 has. - Quota accounting for a container that outlives the request. attach/wait implies a lifetime longer than one HTTP call. Whose quota is charged while it runs, and what reclaims it if the caller dies?
- Is
custom-executor CI actually good enough? Every job stage becomes an RPC round trip. Nobody has measured the overhead against the docker executor on a real pipeline. - Demand. Two packages want it (
gitlab-runner,windmill), and both ship today without it: the runner onshell, windmill with docker-mode jobs simply unavailable. That is a real but not urgent case. Build it when a third consumer appears, or when a paying estate needs per-job isolation.
6. Status of the thing that prompted this
Section titled “6. Status of the thing that prompted this”gitlab-runner is off the socket and installable, without any of the
above. Both declarations are deleted (volumes: entry and the
[runners.docker] volumes string in the config heredoc), the default executor
is shell, and executor: docker without an operator-supplied dockerHost
is refused at install with a message that names this document.
Nothing in this document is required for that, and nothing in this document has been built.