Skip to content
SHC Docs

Swarm start-ordering: should the refusal be revisited?

Archived, superseded by the new line (SHC-1092), which removed Swarm.

Swarm start-ordering: should the refusal be revisited?

Section titled “Swarm start-ordering: should the refusal be revisited?”

STATUS: PROPOSAL. Nothing here is built. It answers one question: the refusal in ce/modules/app/swarm_compat.go blocks 38 catalogue packages from the swarm runtime; is that still the right call, and would an SHC-injected wait-wrapper be an honest way out?

RECOMMENDATION: keep the refusal, reject the in-container wait-wrapper, and build a daemon-side WAVE DEPLOY instead, behind the acceptance bar in §7. See §6 for why the wrapper cannot be made honest and §5 for the swarm-native primitive that changed since the refusal was written.

Every count below was re-measured on 2026-08-11 against shc@0e5621a79 / ce@0796595, docker 29.4.1 (CLI docker/[email protected], engine API 1.54). Claims are marked MEASURED, PROPOSED, or PROOF-OWED.


1. The claim that started this, and the correction

Section titled “1. The claim that started this, and the correction”

The owner challenged a flat statement that 37 of 91 apps cannot run on swarm:

“why are you saying none of my apps support the swarm runtime? I thought our enrichment thing makes most of them actually work correctly on swarm.”

He is substantially right and the flat statement was wrong. SHC does translate. ce/modules/app/swarm_compat.go’s ApplySwarmLoaderCompat rewrites the translatable half of the docker stack deploy reject set and re-runs until the set closes at zero: $-escaping, long-form depends_on whose every condition is service_started → list form, configs.<n>.content → a materialised file:, and a pull_policy drop. Four of the six constructs that used to make a swarm install die at its last step are gone.

What is left is not a missing feature. It is a deliberate refusal of exactly two constructs, and the file argues its own case:

  • service_healthy: “swarm has no start-ordering primitive, so a dependant would start against an unready service”
  • service_completed_successfully: “swarm cannot express run-once-to-completion ordering, so a dependant would start before the one-shot finished”
  • and the design note that governs the whole file: “a translation that quietly weakens a guarantee would replace a loud loader refusal with a stack that deploys green and is wrong, the same defect this file exists to remove, moved one step later.”

That note is the bar any proposal here has to clear. “It deployed green” is precisely the evidence the current design distrusts, so it cannot be the evidence that retires the refusal.


MEASURED. Catalogue at HEAD is 102 packages: 98 under shc/apps/** (three of them fixtures) plus the 4 built-ins that live in ce/builtin/. Not 91. The “37 of 91” figure was stale on both halves.

38 packages are refused on the swarm runtime. Counted independently here (grep for condition: values, then set arithmetic over package dirs) and cross-checked against the pinned ledger swarmUntranslatable in shc/tests/unit/modules/app/catalog_swarm_loader_test.go: the two agree exactly, 38 = 38, with no package on either side only. That agreement is the reason to trust the number.

Blocked byPackages
service_healthy only7
service_completed_successfully only20
both11
Total refused38
  • service_healthy only (7): graphiti, matrix, blockscout, postiz, journiv, kubo, kutt.
  • service_completed_successfully only (20): agent-canvas, dify, librechat, windmill, affine, jellyfin, netbird, frappe, plane, ciso-assistant, defectdojo, defectdojo-classic, fleet, midpoint, midpoint-classic, gitlab-runner, hockeypuck, pmm, traefik, postgres.
  • Both (11): superset, medusa, pulp, mailcow, hyperswitch, keygen, immich, openreplay, supabase, greenbone, wazuh.

The two in bold are ce/builtin/. A swarm-runtime estate cannot stand up its own ingress or its own system store: traefik gates on traefik-init, which writes the static config file traefik’s own --configFile points at; postgres gates on postgres-guard, the fail-closed half of the major-version ladder. That is the sharp end of the whole finding and it is why this is worth solving rather than documenting.

2b. Edge counts, and what a wrapper would have to satisfy

Section titled “2b. Edge counts, and what a wrapper would have to satisfy”

MEASURED, over shc/apps/** (script: parse service blocks, walk every condition: line, resolve the depended-on service, track Jinja {% if %} depth):

  • 47 condition: service_healthy edges. 0 of them sit inside a Jinja conditional; every one is unconditional.
  • 76 condition: service_completed_successfully edges (+2 in ce/builtin = 78 total). 3 sit inside a Jinja {% if %}.
  • 47 of 47 service_healthy targets declare a healthcheck: block in the same file. There is no edge whose dependency has nothing to poll.

That last number is the strongest fact in favour of a wrapper, and it is why the option deserved a real design rather than a dismissal.

2c. How many would an ordering fix ACTUALLY unlock?

Section titled “2c. How many would an ordering fix ACTUALLY unlock?”

Fewer than 38, and the difference matters.

Packages
Refused today38
− also carry platform: (untranslatable for a different reason)−4
− mailcow, which declares compose-only on its own merits−1
− jellyfin, refused first by the host-bind mount policy−1
Genuinely unlocked by a working ordering mechanism32
  • The 4 platform: carriers are blockscout, defectdojo, defectdojo-classic, fleet. MEASURED: platform: never appears alone; all four also carry an ordering condition, so it contributes zero extra refusals but keeps four packages refused after any ordering fix.
  • jellyfin is the one package in both ledgers (swarmUntranslatable and hostBindRefused); the mount policy refuses it before swarm gets a say.
  • The third refusal branch (configs.<n>.content carrying a live ${VAR}) matches 0 catalogue packages, as swarm_compat.go already says of itself.

So: 32, and two of those 32 are the system ingress and the system store. No package in the catalogue is blocked by something other than ordering alone.


3. A sharper problem sitting next to this one

Section titled “3. A sharper problem sitting next to this one”

MEASURED, and not previously recorded anywhere: docker stack deploy silently discards 16 service properties and warns about 2 more. docker/[email protected]’s cli/compose/types.UnsupportedProperties is build, cgroupns_mode, cgroup_parent, devices, domainname, external_links, ipc, links, mac_address, network_mode, pid, privileged, restart, security_opt, shm_size, userns_mode; DeprecatedProperties adds container_name and expose.

The warning is printed by cli/command/stack/loader.go on the deploy path only. MEASURED: docker stack config on a file carrying devices: + privileged: true + network_mode: host prints the keys back out on stdout and emits NOTHING on stderr. SHC’s stack-config preflight (X500062SWMCFG, swarm_driver.go) therefore cannot see this class at all.

Catalogue exposure, service-level keys only:

KeyPackagesReading
restart79benign; swarm’s deploy.restart_policy default is any, and ApplyRestartDefault already skips swarm
expose79benign, informational under compose too
security_opt9sharp, incl. ce/builtin/traefik and ce/builtin/keycloak
devices1sharp, jellyfin (hardware transcode)
shm_size1sharp, gitlab

An app whose security_opt or devices line is dropped deploys green and is wrong, the exact defect class the ordering refusal exists to prevent, currently unguarded and one layer down. This is out of scope for this proposal but should be its own ledger, built the same way swarmUntranslatable was.


4. Option A: the SHC-injected wait-wrapper

Section titled “4. Option A: the SHC-injected wait-wrapper”

The shape asked for: an entrypoint SHC injects into every dependant container that blocks until its dependency is ready, then execs the original command.

Where it would be injected. Not the image (SHC does not build app images) and not an init container (swarm has no such concept; there is no pod). The only seam is the compose render, in the same slot swarm_compat.go already occupies: rewrite entrypoint: to a static shc-wait binary bind-mounted or delivered as a swarm config, with the original entrypoint/command as its argv tail, and the dependency set passed as flags derived from the depends_on block being removed.

What it costs.

  1. A binary in every dependant container. Delivering it as a swarm config is workable (ApplySwarmLoaderCompat already materialises config assets), but it must be static, arch-correct on a mixed cluster, and must not collide with images that have no shell. Entrypoint rewriting also breaks images whose ENTRYPOINT/CMD split carries meaning, and SHC would be guessing that split from the compose file, not from the image manifest.
  2. A new failure mode: the wait that never satisfies. Needs a deadline, and on expiry the honest action is to fail the task, which swarm will then restart, forever, producing a converge timeout that names nothing. That is the original symptom the refusal replaced.
  3. ★ It cannot reproduce either guarantee. This is the disqualifier.

Why it cannot reproduce the guarantee.

  • service_healthy means “the dependency’s own healthcheck has passed”. A wrapper in a different container cannot run that healthcheck; the check is defined against the dependency’s filesystem and process (pg_isready, mas-cli config check, a wget to localhost). The wrapper can only poll a proxy: a TCP connect, or an HTTP probe it invented. A listening port is not health; Postgres accepts connections during recovery, and superset-init fails on a driver set that is installed-but-not-importable. That is a weakened guarantee, deploying green, wrong. Objection unanswered.
  • service_completed_successfully means “the one-shot exited 0”. A wrapper cannot observe another container’s exit code without the daemon API. The catalogue’s one-shots are schema migrations (midpoint ninja.sh, openreplay pg/ch-migration, medusa db:migrate), volume-ownership chowns (agent-canvas, dify), and fail-closed guards (postgres-guard). Approximating “it finished” by polling for a sentinel file or a schema row is per-app, not generic, and the guard case is worse than approximate: a wrapper that cannot distinguish “guard passed” from “guard crashed” turns a fail-closed guard into a no-op, on a live data directory.
  • The only faithful implementation needs the daemon API, and that is already forbidden. Handing every dependant container /var/run/docker.sock is host root across every tenant on the node; the refusal ce/core/volume enforces and the hostBindRefused ledger pins. A wait-wrapper’s honest form is banned by an existing security policy.

Verdict: REJECT. The wrapper is either unfaithful (port-poll) or prohibited (socket). It repeats the defect the refusal was written to prevent.


5. Option B: what swarm can actually do at docker 29.4.1

Section titled “5. Option B: what swarm can actually do at docker 29.4.1”

The refusal was reasoned about swarm’s 2016 answer. Re-measured against the pinned version, three of four facts still hold and one has changed.

ConstructResultEvidence
long-form depends_on (any condition, incl. service_started, incl. required:)REJECTED by the loaderMEASURED: docker stack config → services.web.depends_on must be a list
list-form depends_on: [db]Loaded, then DISCARDEDMEASURED: stack config echoes it, but DependsOn appears nowhere in cli/compose/convert/, and swarm.ServiceSpec (moby api v1.54) has no dependency field of any kind
deploy.update_config.order / --update-order / --rollback-ordernot cross-serviceMEASURED: within-service task replacement order only
deploy.mode: replicated-job / global-jobACCEPTED and CONVERTEDMEASURED: stack config emits it, and cli/compose/convert/service.go:615-626 maps it to swarm.ReplicatedJob / swarm.GlobalJob

★ The last row is the news. Swarm does have a run-once-to-completion primitive now, it is expressible in a stack file at the pinned version, and a job’s terminal task state is complete. swarm_compat.go’s wording (“swarm cannot express run-once-to-completion ordering”) is right about the ordering and now wrong about the primitive. That single correction is what makes Option C possible.

Two further seams, both MEASURED:

  • docker stack deploy --detach=false waits: deploy_composefile.go:73 → waitOnServices → service.WaitOnService → per-service progress until task.Status.State == swarm.TaskStateRunning for every replica.
  • swarm.TaskState carries a distinct starting state ahead of running (moby api v1.54 types/swarm/task.go:26), which is the state a healthchecked container occupies before its check passes.

PROOF-OWED: that swarmkit holds a healthchecked task in starting until the container reports healthy, i.e. that running is health-gated, is documented upstream behaviour but was not re-measured here: swarm is inactive on the dev box (docker info → Swarm: inactive) and this lane does not touch an estate. Option C’s correctness for the service_healthy half depends entirely on this fact, so it is item 1 of the acceptance bar.

PAID, MEASURED 2026-08-14 on the bitspur estate (docker 26.1.5+dfsg1, 3-node swarm, the engine production actually runs, not the dev box’s 29.4.1): a probe service whose healthcheck failed for ~86s was observed Pending → Starting on all 46 consecutive 2-second observations and running on none; its task’s Status.State flipped to running at 20:52:06.971Z, 73ms after the container’s first passing health probe (20:52:06.898Z, from the engine’s own health log). Bar item 1 is GREEN. Evidence: evidence/wave-deploy/bar1-health-gate-bitspur.txt (+ bar1-ANALYSIS.txt) in the 2026-08-14 lane records.


Section titled “6. Option C: the daemon-side WAVE DEPLOY (recommended)”

Move the waiting out of the container and into the one actor that already has a trustworthy view: the SHC daemon. It already runs the deploy, it already has the docker API, and swarm_driver.go already observes swarm convergence.

Mechanism (PROPOSED).

  1. Before ApplySwarmLoaderCompat strips it, read the original long-form depends_on graph and topologically sort the services into waves.
  2. Rewrite each restart: "no" one-shot that is the target of a service_completed_successfully edge to deploy.mode: replicated-job, a translation that strengthens rather than weakens: it tells swarm the truth about what the service is, and gives its completion a terminal state the daemon can read.
  3. Issue N docker stack deploy calls against the same stack name, one per wave. Non-job services are cumulative (waves 1..k). A rewritten one-shot appears in its own wave’s artifact only, MEASURED on bitspur’s docker 26.1.5: an identical redeploy of an unchanged replicated-job puts a second Complete task in the same slot (the job re-executes), while regular services no-op. Carrying an earlier wave’s one-shot forward would re-run a migration while its server boots. Deploy without --prune; absent-from-file means unmanaged, not removed.
  4. Between waves, block on the daemon’s observation: complete for job-mode tasks, health-gated running for healthchecked ones. Not on stack deploy exit, on the task state.
  5. --prune only on the final wave.

Why this answers the objection the wrapper could not. Nothing in the compose file is weakened. The dependency’s own healthcheck stays the predicate (it is the thing swarm gates running on); the one-shot’s own exit code stays the predicate (it is the thing swarm’s job completion records). SHC does not invent a proxy for either. The guarantee is reproduced by deferring the dependant’s creation, which is the same thing compose does; compose likewise cannot make swarm order anything; it simply does not start the dependant yet.

What it costs, honestly.

  • Install stops being one atomic stack deploy. A failed wave leaves a partial stack, though compose already leaves a partial project, and the existing rollback path already handles a partial deployment dir.
  • Update and rollback need the same wave logic or they will race what install carefully ordered.
  • A wave that never converges needs a deadline and a coded error that names the wave and the blocking service. X200074SWMPRE’s shape is the precedent.
  • It does not help the 4 platform: carriers or jellyfin. Ceiling is 32 packages.
  • Job-mode conversion changes how a one-shot appears in shc ps / status roll-ups; deploy_verdict.go’s one-shot folding and swarm_driver.go’s health roll-up both need a job-aware branch, or every migrated stack reports degraded forever.

The refusal stays until all six are green. “It deployed green” is not on this list, deliberately.

  1. The health gate is real. On a live swarm, a service with a healthcheck that fails for 60s must be observed in starting, never running, for that whole window. If this is false, the service_healthy half of Option C is unfounded and only the completed_successfully half survives. (This is the PROOF-OWED item from §5.), GREEN, MEASURED 2026-08-14 on the bitspur estate; see the PAID note in §5.
  2. Observable equivalence, positive. A synthetic app: dependency becomes healthy at T+30s / exits 0 at T+30s; the dependant records the wall-clock instant of its first line of work and the dependency’s state at that instant. Under compose, first_work > dependency_ready. Under swarm-with-waves, the same inequality, over 20 consecutive from-empty runs, zero retries (the standing reliability bar). GREEN, MEASURED 2026-08-14/15 on bitspur after the job-omit fix (v1 failed because wave 3 re-ran init; v3: 20/20 PASS, zero retries). Evidence: evidence/wave-deploy/bar2-waves-20runs-bitspur-v3.txt.
  3. Mutation control. With the wave gate disabled, test 2 must FAIL. A guard that passes with its mechanism removed proves nothing; this house has shipped that defect before. GREEN, MEASURED 2026-08-14 on bitspur: with the wave gate disabled the inequality failed (init_done after first_work). Evidence: evidence/wave-deploy/bar3-mutation-control-bitspur.txt.
  4. The two system builtins, by name. traefik on swarm must serve with a static config whose file mtime precedes the traefik task’s start timestamp. postgres on swarm must be shown refusing a skipped-major upgrade; the guard’s fail-closed behaviour has to survive the translation, not merely run.
  5. The nastiest real app. midpoint on swarm: the server task must not reach running before midpoint-init’s job task reaches complete, proven from task timestamps, not from the stack looking healthy. It is the chosen subject because a midPoint booted against a schema-less database does not fail fast; it sits there retrying, which is exactly the “green and wrong” shape.
  6. The ledger keeps its teeth. swarmUntranslatable shrinks only by packages that passed 2–5 individually. Any package not proven keeps its X400073SWMXLT refusal by name. The ledger is never converted into a permission list.

  1. Keep the refusal. It is correct today and it is the only thing standing between an operator and a silently-wrong swarm estate.
  2. Do not build the in-container wait-wrapper. It cannot reproduce either guarantee without the docker socket, and the socket is already refused for good reason.
  3. Correct swarm_compat.go’s wording on service_completed_successfully: swarm does have a run-once-to-completion primitive (replicated-job, MEASURED at the pinned version); what it lacks is declarative ordering. The refusal is unchanged, its stated reason is not.
  4. Build the wave deploy (§6) behind the bar in §7 if swarm-runtime parity is wanted. The prize is 32 packages including the system ingress and system store, but it is worth nothing if item 3 of the bar is skipped.
  5. Open a separate ledger for §3, the 16 silently-dropped properties. It is the same defect class, it is unguarded, and the stack-config preflight provably cannot see it.

Section titled “9. Related: the requires.runtimes ruling (2026-08-11)”

Landed alongside this proposal, and relevant because it is the field an operator would use to say “this app prefers compose”:

requires.runtimes: is an ORDERED preference list. First entry preferred, later entries acceptable fallbacks; a single entry is a restriction as well as a preference; an app that declares nothing gets the estate default.

MEASURED 2026-08-11: nothing reads the key today: neither requires.runtimes nor the legacy top-level runtimes:, in either repo. Three packages declare one (ivatar [compose, swarm], dependency-track [compose], mailcow legacy [compose]) and four documents claimed install enforced it; those claims are now corrected. The canonical reading is app.RequiredRuntimes / app.PreferredRuntime in ce/modules/app/required_runtimes.go, tested but not yet wired. See app-definition: requirements.