The Docker socket
Layer 6. Services. Services
Some apps start containers of their own. A CI runner starts one per job. An agent starts a sandbox per session. Others manage the containers they ship with: mailcow’s admin UI lists its own services, runs postqueue in its postfix and restarts its dovecot. They do it by talking to the Docker socket, and on a plain Docker host that socket is root on the machine.
On SHC no app container is handed the host’s socket. A deployment that binds it gets a socket of its own, served by shcd, that answers the part of the Docker Engine API these apps call. Every container created through it belongs to the deployment that asked: it carries the deployment’s labels, joins only the deployment’s networks, runs under limits, counts against the node and the tenant’s quota, and goes when the deployment goes. Nothing is passed through to the node’s engine unread.
The app’s own configuration does not change. It still talks to /var/run/docker.sock.
What the render does
Section titled “What the render does”Write the bind the way upstream documents it:
services: runner: image: "{{ vars.images.runner }}" volumes: - /var/run/docker.sock:/var/run/docker.sockIn shc.yaml it is one field of the service’s privileges block:
services: runner: image: "{{ vars.images.runner }}" privileges: docker_socket: trueThe render replaces the source of that bind with the deployment’s broker socket. The target and the mode stay as written, /run/docker.sock is treated the same, and nothing else in the service changes: no DOCKER_HOST is added. The broker socket is a file on the node, one per deployment, and only containers of that deployment have it mounted.
A container that a brokered container creates in turn, by binding the socket again, gets the same deployment’s broker socket.
Your own containers
Section titled “Your own containers”Every deployment’s socket answers its own service replicas on the node, the way a Docker host answers a compose project’s containers, on every runtime alike. The deployment is the compose project: its containers carry com.docker.compose.project set to the deployment’s name and com.docker.compose.service set to the service, beside the labels the compose file gave them, and each is named <deployment>-<service>-<n>. The platform’s own shc.* labels are left out.
| Verb | What it does |
|---|---|
GET /containers/json, GET /containers/{id}/json | Lists and inspects them by id, an id prefix of twelve or more, or name, with the image the compose file names, the state, the health and the start time |
POST /containers/{id}/exec, POST /exec/{id}/start, GET /exec/{id}/json | Runs a command in one, as the user it names, with stdin, stdout and stderr as on the engine and the exit code after |
POST /containers/{id}/start, stop, restart | Starts, stops or restarts its replica through the node’s replica runner. A replica is a pod, so the containers that share it go together. |
kill, pause, wait, logs, attach and a removal of one are refused X403999CBRREP. A container that is not the deployment’s, whether another deployment’s, another tenant’s or one started by hand on the node, is not found by list, id, prefix or name.
A compose network’s names answer under its compose name too, as Docker’s do: <service-or-alias>.<deployment>_<network>, so mailcow’s php-fpm reaches dockerapi.${COMPOSE_PROJECT_NAME}_mailcow-network. An app that filters on its compose project sets COMPOSE_PROJECT_NAME: "{{ stack }}" in its compose file, the deployment’s name.
Self-scoped sockets
Section titled “Self-scoped sockets”A package whose socket is for its own containers only says so in app.yaml:
privileges: docker_socket: selfIts install opens none of the doors below, since nothing is created through it, and its socket refuses everything but the verbs above with X403998CBRSLF. full, the same as saying nothing, is the job-container broker described in the rest of this page. Any other value is refused X400348PRVSCK. mailcow declares self.
What has to be true first
Section titled “What has to be true first”Unless the package declares a self-scoped socket, the install is refused, with the code of the door that is shut, unless all four hold. There is no fallback to the host’s socket.
| Door | How it opens | Refusal |
|---|---|---|
The estate’s licence carries container_runtime | SHC EE, Pro and up | X403984CBREEO on community edition |
This deployment, or its tenant, is granted the socket class | privilege.deployment_grants.<tenant>.<environment>.<deployment> or privilege.grants.<tenant>, set to socket (or a list naming it, or true) at global scope by the master tenant’s operator. The master tenant holds it without one. | X403990HSTGNT |
| The estate has turned the broker on | shc config set container_broker.enabled true at global scope | X403985CBROFF |
| The node’s runtime brokers containers | docker and podman nodes pass each call to their engine. incus nodes run each container as an Incus instance of its own (below). apple nodes broker none. | X403332CBRRTM, naming the node and its runtime |
The first three are asked again on every create, so a grant that is withdrawn stops new containers at once. Containers that already run keep running until they are removed. The telemetry collector’s socket asks none of the three, because it creates nothing (below).
What the socket answers
Section titled “What the socket answers”Docker Engine API versions 1.40 to 1.51, with or without the /vX.Y prefix. A version outside that range is refused X400991CBRVER.
| Area | Endpoints |
|---|---|
| Daemon | GET and HEAD /_ping, GET /version, GET /info |
| Images | POST /images/create (a pull), GET /images/{name}/json |
| Containers | POST /containers/create, GET /containers/json, GET /containers/{id}/json, POST start, wait, stop, restart, kill, pause, unpause, attach, GET /containers/{id}/logs, DELETE /containers/{id} |
| Exec | POST /containers/{id}/exec, POST /exec/{id}/start, GET /exec/{id}/json |
| Volumes | POST /volumes/create, GET /volumes, GET /volumes/{name}, DELETE /volumes/{name} |
| Networks | POST /networks/create, GET /networks, GET /networks/{id}, DELETE /networks/{id}, POST connect, disconnect |
That is what GitLab Runner’s docker executor, Woodpecker, Drone and OpenHands call. Attach and exec are hijacked streams, as on the engine. Everything else (build, commit, events, archive, prune, system df, swarm, plugins) answers 501 with X501992CBRNSV naming the endpoint.
The deployment sees only what it made and its own replicas. List endpoints return its own containers, networks and volumes. Any other id, whether another deployment’s container or replica or something started by hand on the node, answers 404 as an absent object does and is not touched.
Images work the same way. The node keeps one image store for everyone, so a deployment can start and inspect only images it pulled through its own socket. Anything else answers the engine’s own No such image, which every client meets by pulling.
On an incus node
Section titled “On an incus node”An incus node has no engine to pass calls to, so the broker runs each container it creates through the node’s incus executor, the one that runs the deployment’s own replicas. The socket answers the same endpoints, with the same doors, refusals, labels, limits, rows and quota, and a client sees no difference in what it sends.
Each container is a compose project of its own on the node, shcbrk-<id>, with one entry. A create writes it and starts nothing, as on the engine. The first start brings it up as an Incus instance, with the executor’s translation of every field the broker passed: the environment is staged in memory and never written into the instance’s config, the image is pulled with the tenant’s registry connections, and the container joins the deployment’s networks and the tenant’s fabric bridge at its pool address as a replica’s entry would. A later start starts the same instance. A remove takes the project down with its anonymous volumes. An Incus instance holds every capability its user namespace grants, so the broker drops each one an engine would not give a container: a brokered container holds the same capabilities on incus as on docker.
A pull goes into a project of the deployment’s own on the node, shcbrk-<deployment id>, which only its pulls use. A container’s project made after it finds the image there by its fingerprint and downloads nothing again. The pull’s stream reports progress every few seconds while Incus pulls, so a client’s read timeout does not end it. The deployment’s uninstall takes that project down with the rest.
Attach is served by the container itself. The exec helper that starts a brokered container’s first process listens on a socket in the container’s status directory, and the broker connects to it from the node once the container runs. What the client writes is the container’s stdin when its create opened one (OpenStdin, and StdinOnce ends it with the first client), and what it reads is stdout and stderr apart, framed as the engine frames them, while the console keeps getting both. An attach made before the start waits for it, and what the container writes before the first client reaches the socket is kept for that client. That is how GitLab Runner’s docker executor runs each stage: attach, start, the script on stdin, the trace to the container’s end, then the same container again for the next stage. Its service containers are reached the way it reaches them on the engine: inspect reports a running container’s address on the tenant’s fabric, and the ExtraHosts the runner passes beside a job container’s Hostname go into the hosts file the node stages for the container at its fabric address.
host networking is the caller’s on incus too: the container joins the network namespace of the deployment’s container that holds the broker socket, through liblxc’s lxc.namespace.share.net, with none of the network devices its profiles would give it, and with that container’s resolvers. Each start of the container joins the holder’s namespace as it is then, so a restart of the holder is followed. OpenHands’ sandboxes run that way and listen on the OpenHands container’s own localhost. none gives the container a namespace of its own with loopback alone.
What differs from the engine:
| On incus | |
|---|---|
Stats, events, attach to a Tty container | 501, X501992CBRNSV. Exec runs through the executor, with no tty. |
| Logs | The instance’s console: stdout and stderr together, as stdout. The bytes are the container’s as it wrote them, as on the engine, and a Tty container’s are as a terminal writes them. since, until and timestamps are refused X400796EXELOG. |
| A client’s registry credential | Refused X403987CBRFLD naming X-Registry-Auth. The tenant’s registry connections are what the node pulls with. An empty one is no credential. |
| Networks after the first start | Fixed. A connect or disconnect once the container has an instance is refused X403987CBRFLD. |
ExtraHosts host-gateway | Refused: the address of the holder on a shared network is not read on incus. |
| A container’s name on its networks | Not answered: the node’s resolver answers deployments’ names, not a brokered container’s. |
| A field the incus runtime has no translation for | Refused at create with the executor’s own code, X400776INCKEY, naming the field and why. |
The telemetry collector’s socket
Section titled “The telemetry collector’s socket”One deployment’s socket is different. The system telemetry collector, master/default/collector, gets a read-only view of every container on its node, so it reads container stats and logs through its socket the way it would read the engine’s. The view belongs to that deployment by its tenant, environment and stack. No setting gives it to another deployment, and a stack called collector anywhere else gets the ordinary socket.
Its render opens none of the licence, grant and opt-in doors, since nothing is created through it. The node’s runtime still has to broker containers.
The node reads through the same view. A Docker node’s own collector project, the one that tails container logs and reads host metrics on the node, binds this socket read-only where docker_stats dials, in place of the engine’s socket. No SHC project binds /var/run/docker.sock.
The daemon serves the view socket from boot on every node that brokers containers. On an incus node it answers the listing, inspect and logs, and stats and events answer 501. It stays open whether or not master/default/collector has a replica on the node, and through an uninstall of that deployment. When the daemon has to make the socket again, after a stop and a start or a reboot, the node’s collector container is made again on its next sweep, because its bind still holds the old socket.
A Mac running Docker binds none. A socket on the Mac’s filesystem does not cross into the engine’s Linux VM, so that node’s container stats are not collected.
| Endpoint | What it answers |
|---|---|
GET and HEAD /_ping, GET /version, GET /info | As on every broker socket |
GET /containers/json | Every container on the node: id, names, image, labels, state, status and ports. The command, mounts and networks are left out. |
GET /containers/{id}/json | Any container on the node: id, name, image, labels, hostname, state and resource limits. The environment, command, binds, mounts, host paths and healthcheck output are left out. |
GET /containers/{id}/stats | The engine’s stats of any container on the node, once, one-shot or streamed |
GET /containers/{id}/logs | The log of any container on the node |
GET /events | Container lifecycle events only: create, start, restart, stop, pause, unpause, update, rename, kill, die, oom and destroy. Exec events and every other type are left out, whatever the client filters for. |
Labels and event attributes leave out compose’s com.docker.compose.project.* labels, which name the project’s directory and files on the node. Logs are the one place another tenant’s data crosses. Every other endpoint, every write included, answers 501 with X501992CBRNSV and the engine is not asked. An {id} that is not a container id or name answers X404987CBROBJ.
What a created container gets
Section titled “What a created container gets”| Labels | shc.tenant, shc.environment, shc.stack, shc.node, shc.job=broker, a broker id, and the client’s own labels. A client label in the shc., com.docker.compose., com.docker.swarm., com.docker.stack. or io.podman.compose. namespace is refused. |
| Networks | The deployment’s own networks of its replicas that hold the socket, and networks the deployment created through the broker. Those are bridge networks, always internal. bridge, default and a request that names no network mean the deployment’s network, never the engine’s default bridge. On the SHC fabric a created container with a network of its own also joins the tenant’s fabric bridge, as its default route, at an address filed under the deployment, so it reaches what the deployment reaches and asks the node’s DNS first. The node’s DNS answers it every name the deployment calls its providers by, including a name a plug gives to only some of the deployment’s services, so a job a runner starts reaches its GitLab by the name the runner uses. Nothing reaches it at that address, and inspect and list leave that network out. The addresses are a pool the deployment keeps on the node, at most container_broker.max_containers of them: each is held by one container from its create to its removal, then reused by the deployment’s next, and the pool is released when the deployment leaves the node or is uninstalled. host means the caller’s host: the container shares the network namespace of the deployment’s one container on the node that holds the broker socket, never the node’s. none, or NetworkDisabled, gives the container no network. |
| Volumes | Named volumes the deployment created through the broker, and tmpfs. The Docker socket again, which is the same broker socket. A host path is refused. |
| Limits | The request’s CPU and memory, or the estate’s defaults when it names none. A pids limit, always. |
| Names | The client sees the names it chose. On the node, containers, networks and volumes are prefixed shcbrk-<id>-. A container is still answered by the name its client gave it on every network of the deployment it joins, as Docker answers a container’s name on a user-defined network. It is not answered by that name on the tenant’s fabric bridge. |
What it refuses
Section titled “What it refuses”A create is refused before anything exists, in the Docker error body, so the runner prints the reason. X403987CBRFLD names the field.
Refused outright: Privileged, CapAdd, Devices, DeviceRequests, DeviceCgroupRules, SecurityOpt, Sysctls, a non-default Runtime, PortBindings and PublishAllPorts, any PidMode, IpcMode, UTSMode, UsernsMode or CgroupnsMode, MaskedPaths and ReadonlyPaths, a host path in Binds or a bind in Mounts, a restart policy other than no, a log driver other than json-file or local.
Refused unless the other object is the deployment’s own: NetworkMode: container:<id>, VolumesFrom, Links.
A link gives the linking container the alias as a name for the linked one, on the network the create names. That is what docker run --link gives on a user-defined network, and it holds whether the client sends the link in HostConfig or on the network’s endpoint. An alias has to be a host name. The link environment variables of Docker’s default bridge are not set, since no created container is on that bridge.
NetworkMode: host is refused when the deployment has no single running container on the node holding the broker socket. A container that shares a network namespace, with host or container:<id>, may not set Hostname, ExposedPorts, Links, DNS or ExtraHosts. A container with no network may not set Links.
ExtraHosts may name only addresses on the deployment’s networks. host-gateway resolves to the address of the deployment’s container that holds the broker socket, on the network it shares with the created container, and is refused when that is not one address.
Any other HostConfig field is refused when it is set. The list of fields that may be set is short on purpose: Binds, Mounts, NetworkMode, RestartPolicy, AutoRemove, LogConfig, ShmSize, Tmpfs, Init, ReadonlyRootfs, CapDrop, GroupAdd, DNS, DNSSearch, DNSOptions, ExtraHosts, OomScoreAdj, VolumesFrom, Links, ConsoleSize, and of the resources Memory, MemoryReservation, MemorySwap, NanoCPUs, CPUShares, PidsLimit and Ulimits.
A brokered container reaches its caller over the shared network, or on its caller’s own localhost when it asked for host, never through a port published on the node.
| Code | Meaning |
|---|---|
X403987CBRFLD | A field the broker does not allow, named |
X403989CBRLIM | The deployment is at its cap of containers, networks or volumes |
X403994CBRFIT | The node has no room for the container’s CPU or memory |
X403001CPUEXC, X403002MEMEXC | The tenant’s CPU or memory quota is spent |
X404987CBROBJ | No such object of this deployment |
X403998CBRSLF | The package declares its socket for its own containers only, and the request asks for more |
X403999CBRREP | A verb the socket does not carry out on the deployment’s own replicas |
X400984CBRREQ | The request is missing a field or cannot be decoded |
X400991CBRVER | An API version the socket does not answer |
X501992CBRNSV | An endpoint the socket does not serve |
X503988CBRENG | The node’s engine or the cluster store cannot be used right now |
X503989CBRNOD | The deployment has no replica on this node |
X409996CBRNET | No network of the deployment’s own on this node is one the container may join |
X409993CBRSCK | Something that is not a socket sits where the broker socket goes |
X503986CBRNYI | shc.container.run was authorised on a daemon that carries no executor for it, and nothing ran |
How the containers are counted
Section titled “How the containers are counted”Every brokered container has a row in the cluster store from before the engine is asked until the container is removed. The row names the deployment, the node, the container, the image and a digest of its command.
The node’s room is read from those rows. A brokered container takes its CPU and memory limits off the node it runs on, the same room replicas are placed in, so the scheduler does not bind a replica onto memory a CI job holds. It is never counted as outside use. A create the node has no room for is refused X403994CBRFIT. Nothing is evicted to make room.
The tenant’s quota is read from them too. The limits of a deployment’s brokered containers, on every node, are charged to its tenant and its environment beside what the deployment itself holds. A create that would pass a cap is refused with the cap’s own code.
A container that has exited or died holds no room and no quota. Its limits come off the node and off the tenant’s quota at the node’s next pass, within 30 seconds of the exit, so a finished job does not keep the next one out. It still counts as one of the deployment’s container_broker.max_containers and keeps its address until it is removed. Starting it again is admitted like a create: the node must have room for it and the quota must cover it. A start that fails either is refused X403994CBRFIT or with the cap’s code, and the container stays stopped.
shc job ps lists them with the running jobs, each under its deployment as the job broker, with the engine’s id for the container. One created and not yet started is pending; one that exited is no longer listed there. The state is the one the node’s engine listed at the node’s last pass, which runs every 30 seconds, and a container made since that pass reads running:
shc job ps -s runnershc inspect lists them under the deployment’s containers too, under the service broker, by the name each client gave its container.
Create, start, kill and remove each write an audit event naming the deployment, the container, the image and the command digest.
When they are removed
Section titled “When they are removed”- When the client removes them, or they were created with
AutoRemove. - When the deployment is uninstalled: on every node, its brokered containers with their anonymous volumes, the networks and volumes it created through the broker, their rows and the socket.
- When the deployment has had no replica on a node for a minute, after a move or a scale down. The node clears what the broker made there within two minutes of the replica leaving.
- When a container has not run for
maintenance.stale_jobs_retention_hours, 24 by default: one that exited that long ago, or was created that long ago and never started. A client that is still there removes what it has finished with, so one left that long belongs to a client that died, such as a runner killed in the middle of a job. Until it goes it holds one of the deployment’scontainer_broker.max_containers, and one that never started holds its room and its quota too. Its anonymous volumes go with it. Volumes and networks the deployment created by name stay. The removal is audited like the client’s own. - When a container still runs and its client is gone. That is the case when every container of the deployment that holds the broker socket on the node started after the container was made, no request has named or listed the container since, and that start is older than
maintenance.stale_jobs_retention_hours. The usual one is the service container of a job whose runner was restarted in the middle of it: the build container exits, the service runs on. A client that is still there shows what it holds. An inspect, a start, a wait, a log read, an exec or a removal of the container is enough, and so is a listing that returns it, and a container named or listed since the holder started is kept for as long as that holder runs. So an app that restarts and takes its running containers up again keeps them, a manager that polls the list keeps everything it shows, and a client that never looks at a container again loses it after the retention. The removal is forced, takes the container’s anonymous volumes and is audited with the reason.
When a node cannot remove everything the broker made for a deployment, it logs the warning X200995CBRSWP and tries again within a minute.
A restart of shcd breaks neither. The socket’s listener is handed back by systemd, so a running container keeps a working socket. The rows are in the store, so the first pass after a restart removes a brokered container that has no row and deletes a row whose container is gone.
A full stop and a later start of shcd is different, because systemd keeps no listener for a stopped service. The new shcd serves each socket again for every deployment that has a running replica binding one, and restarts a replica whose container still holds the old socket, once, with the members of its pod together.
Settings
Section titled “Settings”All of them are global scope only and take effect on the next request.
| Key | Default | Meaning |
|---|---|---|
container_broker.enabled | false | Turns the broker on for the estate |
container_broker.default_cpus | 1 | CPUs for a container whose create names none |
container_broker.default_memory | 1073741824 | Bytes of memory for a container whose create names none |
container_broker.pids_limit | 2048 | The most processes a brokered container may hold. A create may ask for fewer. |
container_broker.max_containers | 16 | Brokered containers one deployment may have on a node |
container_broker.max_networks | 8 | Networks one deployment may have created through the broker on a node |
container_broker.max_volumes | 64 | Volumes one deployment may have created through the broker on a node |
See also
Section titled “See also”- What SHC does to your compose file: every mount source and what the render does with it
- shc.yaml: the
privilegesblock - Config keys reference