Frappe image architecture: baked apps, dynamic apps, and both at once
Status: open design. Baked apps already ship. Installing an app into the image without a rebuild is still the proposal.
The owner’s objective, stated verbatim: the SHC Frappe image must work for vApps (“where you can install an app dynamically instead of them having to be baked into the images”) and support images with apps baked in, and support both, so a builder can bake some apps (Keycloak, because the OIDC plug needs it) while others are still added afterwards.
The open technical question was whether an app can be built once into a tarball and installed without a rebuild, or whether the target must clone source and build every asset and pip-install everything, or some mix.
Both halves are answerable, and they are independent of each other. That is the central finding: build cost and restart cost are separate problems with separate solutions, and conflating them is what made this look like one hard problem.
Evidence: /var/tmp/frappe-arch/0{1..5}-*.md. Every number below is MEASURED on a
throwaway bench built from the pinned image, not inferred.
1. The answer in one table
Section titled “1. The answer in one table”| Objective | Verdict | Mechanism |
|---|---|---|
| Baked apps | Already shipped | bakeApps@boolean in app.yaml, install-time only, guarded twice |
| Dynamic app install without rebuild | YES | Prebuilt artifact: no clone, no npm/esbuild, no compiler, no network |
| Dynamic install without downtime | YES, with one build-time line | A directory pre-seeded onto sys.path at build time |
| Both modes at once | YES, they are orthogonal | The hot-load directory does not care what was baked |
1b. ★ THE DEPLOYMENT MODEL: RULED BY THE OWNER, 2026-08-05
Section titled “1b. ★ THE DEPLOYMENT MODEL: RULED BY THE OWNER, 2026-08-05”Everything below this section is measurement. This section is the contract those measurements have to serve, and it was ruled after them.
The governing principle
Section titled “The governing principle”The image is an INPUT, not something SHC manufactures.
SHC must work with whatever image it is handed, carrying whatever packages that image happens to have, and must know what that image can and cannot do. It does not impose a blessed image and it does not build one on the fly.
★ A build sidecar was proposed and REJECTED. (Run the fat image as a one-shot builder, produce the artifact, install it into the slim runtime.) It is technically sound (same base ⇒ ABI-compatible by construction) and it is still the wrong shape, because it manufactures a capability the operator did not ask for and hides which image is actually in play. Do not reintroduce it.
Two axes, deliberately independent
Section titled “Two axes, deliberately independent”Where an app comes from: baked into the image · added dynamically as a vApp · both at once. The third is the case that must work; it reads as hard only while the axes are conflated.
What the image can do: carries build tooling (compilers, headers, node/yarn) · does not. A lighter image is a legitimate, first-class choice.
The supported matrix
Section titled “The supported matrix”| external app arrives as | image has build tooling | result |
|---|---|---|
| prebuilt artifact | irrelevant | installs (ABI must match, §5) |
| source (git ref) | yes | installs, compiles on the target, today’s path |
| source (git ref) | no | REFUSES |
| tar | detect what is in it, then as above | as above |
Baked apps work in any image, because they are already present.
The owner’s rule, verbatim: “it work with any image that comes with whatever packages they come with as well as work with external packages. however if the external packages are not pre-compiled you have to use an image that comes with the build stuff.”
What “prebuilt” means, and the asymmetry inside it
Section titled “What “prebuilt” means, and the asymmetry inside it”The two halves of an artifact are not alike, and the difference decides what may be pinned loosely and what may not.
The node/frontend half is compiled in, fully. It is frontend output; there is no
reason to run a bundler on a production host. And it is platform-independent; the
built assets are just files, and bundle hashes were measured deterministic across
independent builds. This half travels anywhere. It is also where the cost is:
bench get-app --skip-assets does not skip yarn install (38 s of 43.7 s), and it
leaves 680 MB of node_modules + 93 MB of .git in every production bench.
The python half ships as wheels, and a python install step REMAINS, but it degrades from a build to a link:
pip install --no-index --find-links=<bundled-wheels> … # deps: offline, no compilerpip install -e apps/<app> --no-deps # the app: writes a .pth★ The asymmetry: the python half is ABI-locked where the node half is not. That is the whole reason §5’s pin is the base image digest and not a python version.
Neither half makes bench install-app cheaper: 24.8 s from an artifact vs 25.9 s
from a clone. Database work is the floor.
Detection
Section titled “Detection”- A git ref is source by definition. No probe needed.
- A tar must be SELF-DESCRIBING: a manifest inside declaring
{app, version, commit, base_image_digest, abi_tag, has_assets}. “Is this prebuilt” is really “does this carry built assets and wheels for this runtime”, which cannot be answered by sniffing forpublic/distand a wheels directory. It is also the same file the ABI check needs, so it costs nothing extra. - Image capability must be PROBED, not declared (a compiler on PATH, python headers, node/yarn for the asset build) and recorded on the deployment rather than re-probed per install. Honesty note: “has build stuff” is not strictly binary, since a source install can succeed without a compiler when every dependency ships a manylinux wheel. Require the toolchain anyway and offer an explicit override; the alternative is attempt-and-discover, whose cost is documented in §4.
★ Where the refusal happens: the load-bearing part
Section titled “★ Where the refusal happens: the load-bearing part”At RESOLVE time, before anything mutates.
This is the entire lesson of the incident in §4: a failed vApp install left a production
site 500ing on every route because the app reached apps.txt before anyone knew it
could not work. When the source type plus the image probe are enough to know “this
image cannot build this app”, SHC must refuse having changed nothing, which is
strictly better than the transactional rollback that now exists to clean up after the
attempt.
What follows from the model
Section titled “What follows from the model”- SHC does not need to ship a frappe image at all for the vApp story. The
baked-
frappe_keycloakimage is a convenience for the OIDC plug, not a structural requirement. - The factory is not a dependency; see §7.7.
2. The tarball question, settled
Section titled “2. The tarball question, settled”An app builds once in a builder and installs offline. Measured, --network none:
| App | Clone + build (current) | Artifact (offline) |
|---|---|---|
frappe_keycloak | 3.4 s, 896 KB on disk | 1.63 s, 356 KB (tarball 837 KB) |
hrms v15.36.0 | 46 s, 793 MB on disk | 1.08 s, 19 MB (tarball 9.1 MB) |
bench install-app hrms from the artifact: 24.8 s vs a 25.9 s clone-path control,
parity, because that step is database work either way. /app returned 200 with all
five prebuilt hashed bundles and no bundler on the machine. Bundle hashes are
deterministic across independent builds.
Approach A (full pre-build: source + built assets + wheels) wins. Building assets
at install time is the worst option: it keeps node, the network, and the upstream
frappe-ui/vite failure mode on every target instead of once in a builder.
Two facts that make the builder pattern correct independent of everything else:
bench get-app --skip-assetsdoes not skipyarn install. 38 of its 43.7 s were yarn, and it leaves 680 MB ofnode_modulesplus 93 MB of.gitin every production bench.frappe_keycloakis a misleadingly easy case. Itspublic/holds exactly one file:.gitkeep. Nopackage.json.bench buildemits an empty file table in 83 ms. Nothing about assets can be learned from it; every asset finding above ishrms-derived.
3. The restart question, settled, and the two findings that look contradictory
Section titled “3. The restart question, settled, and the two findings that look contradictory”Two investigations reached opposite-sounding conclusions. They are both correct and they compose, because they tested different mechanisms.
A new .pth requires a restart. Both investigations agree, independently.
uv pip install -e writes a plain-path .pth, and site.py processes .pth files
once, at interpreter startup. Measured: after a successful offline install +
install-app, every route returned 500 ModuleNotFoundError while a fresh
interpreter imported the app fine. SIGHUP does not help; it re-forks workers from
the same master, whose sys.path was fixed at boot (pids 12,13 → 235,236, still
raising). SIGUSR2 does work but is unusable: gunicorn is PID 1, so retiring the old
master kills the container.
(Correction for the record: the Dockerfile says pip install -e leaves an
.egg-link. Modern pip/uv write a .pth. The conclusion is unchanged; both are
startup-only, but the file it names does not exist.)
A directory already on sys.path at boot can receive new packages hot. This was
never tested by the tarball investigation. One build-time line:
mkdir -p /home/frappe/frappe-bench/hotappsprintf '/home/frappe/frappe-bench/hotapps\n' > env/lib/python3.11/site-packages/zz-shc-hotapps.pthThen at runtime, symlinking an app’s package into hotapps/ makes it importable to
every already-running worker, with no restart and no signal. Proven end to end
with the real payments app on a never-restarted fleet: get-app → symlink →
install-app, 7.1 s total, zero downtime, 10 DocTypes created, /login 200.
So: the artifact removes the build; the pre-seeded directory removes the restart. Neither alone is sufficient; together they are.
The one race
Section titled “The one race”Directory mtime has a 4 ms quantum (CONFIG_HZ=250), and CPython’s FileFinder
rescans a path directory only when mtime changes. Same-instant hot-adds went stale
23 times in 25. importlib.invalidate_caches() or any gap > 4 ms → 15/15 clean. In a
real flow install-app supplies 1–3 s of gap, but it is a race, not a guarantee: the
airtight form calls invalidate_caches() per worker, or creates the symlink first
and lets install-app provide the gap.
4. What the outage actually was: BOTH of the Dockerfile’s premises were wrong
Section titled “4. What the outage actually was: BOTH of the Dockerfile’s premises were wrong”(Both have since been corrected in place. The conclusions survive as deliberate policy; only the stated mechanisms were false.)
Premise two, for completeness: VOLUME does not stop the write. The Dockerfile
argued it cannot register the app because sites/ is a declared volume and Docker
discards build-time writes to it. Re-measured on the same base: true only on the
deprecated builder (DOCKER_BUILDKIT=0, where the append vanishes at zero exit),
and false under buildx/BuildKit, which is exactly what the images job runs,
where it persists into the layer, the image, and a fresh named volume. So the
restraint is a policy this project holds and a test enforces, not a wall Docker puts
up. apps/ is not a volume under either builder.
frappe/docker/Dockerfile states: “Frappe imports every app in apps.txt on every
request, so the site 500’d on EVERY route.” Measured false. Frappe consults
three lists with three filters:
| Consumer | List | Source |
|---|---|---|
frappe.init() outside a request, CLI, bench, scheduler | get_all_apps(with_internal_apps=True) | sites/apps.txt |
module map + _load_app_hooks() | get_installed_apps(_ensure_on_bench=True) | DB ∩ apps.txt |
| website template resolution | get_installed_apps() raw | DB only |
An app in apps.txt that is not importable leaves HTTP entirely green
(/api/method/ping 200, /login 200, /app 301) while bricking the whole bench
CLI, including bench clear-cache, so you cannot clear your way out.
The outage came from the database list via the website renderer, not apps.txt.
This inverts which step is dangerous:
bench install-appis the risky step. Theapps.txtline is comparatively benign.
It also explains the recovery: deleting the apps.txt line restores the CLI
instantly, but /login keeps 500ing, because the renderer walks the raw DB list:
only restoring the code or uninstall-app fixes it.
Therefore the safe order is: code first, registration second. Never the reverse.
The Dockerfile’s conclusion (bake frappe_keycloak) remains right. Its stated
reason does not survive measurement and should be corrected in place.
5. ★ The constraint that outranks all of this: ABI, and the real pin
Section titled “5. ★ The constraint that outranks all of this: ABI, and the real pin”An artifact cannot cross the 3.11 → 3.14 hop. The bench venv holds 70 native
.so files, 62 ABI-locked to cpython-311. On real python:3.14-slim,
cp311-cp311 wheels are refused at the filename. abi3 is a narrower escape than
it looks: cryptography’s abi3 wheel installs on 3.14 and is still unusable because
its dependency cffi is cp311-only.
frappe_keycloak looks immune and is not. All 15 of its wheels are pure-python,
but frappe_keycloak → python-keycloak → jwcrypto → cryptography → cffi hits the
wall. It is clean only because it delegates its native dependency to the base image.
The real pin for an artifact is the BASE IMAGE DIGEST, not the Python version. An
abi_tagguard is necessary but insufficient.
Compounding it: SHC pins v15.118.2, which predates frappe_docker’s April-2026
asset redesign. SHC is on the old asset contract and inherits the new one only on
the v16 move, so the interpreter migration and an asset-layout migration arrive
together.
6. The delivery hazard that is live right now
Section titled “6. The delivery hazard that is live right now”The image is built and pushed, and nothing pins it. docker/docker-bake.hcl
builds …/shc/frappe on every main push (five tags published), while
apps/frappe/vars.yaml still names upstream frappe/erpnext:v15.118.2. The baked
frappe_keycloak the OIDC plug depends on exists on no estate.
And switching the pin may deliver nothing. compose.yaml seeds the bench volume
gated on [ "$BAKE_APPS" != "yes" ] && [ ! -f /bench-init/.seeded ]. In the default
mode apps/ is a volume seeded once, never refreshed, so an image swap is a
silent no-op for the app tree. Only bakeApps=yes puts apps/ in the image layer,
and bakeApps is install-time only.
Independent corroboration: frappe_docker rejected a shared apps volume for exactly this reason (issue #790; a named volume is seeded from the image only on first creation, freezing app versions). Two teams reached this from opposite directions.
Also: bench restart is a silent no-op in containers.
restart_supervisor_processes returns immediately when supervisorctl is off PATH,
with no warning and no error. Nothing in-container can pick up a new app; only an
orchestrator-level restart, or the hot-load path in §3.
7. ★ OPEN QUESTIONS FOR THE OWNER
Section titled “7. ★ OPEN QUESTIONS FOR THE OWNER”-
Do we adopt the hot-load directory? It buys zero-downtime dynamic installs for one build-time line. Cost: a
sys.pathentry that is writable at runtime, and a 4 ms race needinginvalidate_caches(). Restart is cheap (687 ms to first 200), so this buys availability, not speed. Recommendation: yes; availability during app install is exactly the vApp promise. -
Artifact format: ★ PARTLY RULED 2026-08-05, see §1b. The artifact is Approach A and carries a self-describing manifest
{app, version, commit, base_image_digest, abi_tag, has_assets}; resolution refuses on base-image mismatch rather than installing something that imports in the venv and 500s on request. Still open: where it is built and published. Note this is now a smaller question than it was; under §1b the image is an input, so SHC is not obliged to build artifacts for apps it does not ship. An operator pointing at a git ref with a build-capable image needs no artifact at all. -
Retention.ANSWERED 2026-08-05: safe today, with a named landmine. The G19 rot is in the generic package registry. The image lives in the container registry, whosecontainer_expiration_policy.enabled = false; empirically nothing has been pruned (forward-auth retains 534 commit tags, keycloak 270, traefik 269), and the three pinsce/builtinalready carries still resolve. The landmine: the disabled policy readskeep_n: 10, older_than: 90d, name_regex: .*with noname_regex_keep; enabling it as written would delete this pin and the three ce ones together. Do not enable it without an exclusion. -
How does a plug declare a host-side prerequisite? The OIDC plug needs
frappe_keycloakpresent. Today that is solved by baking, and there is no way to state it; the consumer’s only phase ispost_deployment, which runs after the consumer is live, i.e. after the failure window. This is G1’s cousin. -
Does an image record what it carries? The vsocket’s refusal is mode-shaped (
bakeApps=yes→ refuse everything) rather than set-shaped, because the fact does not exist in readable form. “Bake some, add others” needs it to. -
Existing deployments: STILL OPEN, and now measured. The pin landed (
apps/frappe/vars.yaml→…/shc/frappe:76624a36@sha256:114137d9…, verified by pulling and probing the image:apps/carries frappe, erpnext andfrappe_keycloak, and./env/bin/python -c "import frappe_keycloak"resolves). The delivery matrix is now confirmed rather than suspected:case does frappe_keycloakarrive?new install, bakeApps=true✔ apps/is the image every bootnew install, bakeApps=false✔ volume created empty, seeded from this image existing deployment, bakeApps=false, i.e. both live benches✘ silent no-op The seed is keyed on
[ ! -f /bench-init/.seeded ], never on image identity. The pin switch must not be read as a delivered fix forbit/ffx. One thing improved though: the bump does deliver everything outside the bench mount, including the recorded manifest, so the drift job now reportsstaleand names the once-only seed as the cause, making the no-op visible instead of silent.Do we accept new-installs-only, add an image-keyed re-seed, or require a
bakeApps=truereinstall? This is a consented estate decision, not an engineering one. -
Does SHC’s frappe image come from the factory?ANSWERED 2026-08-05: no, and under §1b it does not need to. Measured rather than assumed:- SHC’s image is
FROM frappe/erpnext:v15.118.2@sha256:2372e19…, straight upstream. The factory (~/frappe/factory, a different repo tree entirely) is not in SHC’s image lineage today. - The factory does already implement the fat/slim split, and implements it well:
base = python:${PYTHON_VERSION}-slim-${DEBIAN}+ runtime deps →builder = base+ build deps (build-essential, gcc, libffi/libmariadb/libpq/ libsasl2/libldap2/libtiff5-dev …) →frappe = FROM builder AS b; FROM base; COPY --from=b /home/frappe/frappe-bench. It compiles in the fat image and ships on the slim one. - ★ It also carries
ARG PYTHON_VERSION=3.11.6, a parameterised interpreter, which is the only lever anyone has for the v16 wall (§5: frappe v16 requires ≥3.14; upstream ships 3.11.6).
Under §1b this is a convenience, not a dependency. The factory is simply one source of images where
builderhas the toolchain andbasedoes not. Adopting it as SHC’s base remains a live option with a real trade-off: it buys control of the interpreter (and therefore a v16 path that does not wait on upstream), and it costs divergence from the upstream image the community survey identified as the supported path. Not required by anything in this document. - SHC’s image is
8. Defects found along the way (tracked separately, not fixed here)
Section titled “8. Defects found along the way (tracked separately, not fixed here)”- ★
sites/apps.json, wherebench get-apprecordscommit_hash/branch/version, is never maintained by SHC’s vsocket. Combined with the builder’srm -rf apps/$_NAME/.git, that is the whole of G17 (UNVERSIONED). - The builder soft-fails clones (exit 0): an image can ship green missing a declared app.
bench get-appis not atomic (leaves a.pth+apps.txtentry when the build step fails) and scramblesapps.txtorder.- Neither the baked
apps.txtnor bench’s rewrite ends in a newline, soecho x >>silently corrupts the last entry (measured: a literalprobeappghostapp). ls -1 apps > sites/apps.txtwrites alphabetical order (erpnextbeforefrappe). Harmless today,get_all_apps()forcesfrappefirst, but it clobbers deliberate ordering.- If an app is absent from
apps.txtbeforebench build, esbuild dies aspath.resolve(undefined)with exit 143, which reads like SIGTERM/OOM and is not. - The bench-init seeding loop is not idempotent.
9. What this does not claim
Section titled “9. What this does not claim”The prebuilt path was proven for frappe_keycloak (trivial, no assets) and hrms
(real, asset-heavy). It was not proven for an app with native extensions of its
own; that case still needs the base-image pin of §5 to be enforced, not just
recorded. And one install-app hrms failed once with a DocType-sync error that did
not reproduce across two subsequent clean runs with the same artifact; the clone-path
control also succeeded. Recorded as unattributed rather than explained away.