Skip to content
SHC Docs

Integration teardown: wake the 22 dead jobs, then collapse three runners into one

Status: open design. Steps 3, 5, and 7 are not done.

Revision 4 (status only, 2026-08-05). Steps 4 and 6 LANDED on ce main 2026-08-04: 988d4c2 (PersistedEdgeState written at bind), 63e44b6 (the uninstall leg runs jobs.unintegration against the frozen edge), 45aae15 (the -i <ref>! disconnect leg, snapshot-before-deploy / run-after), 1c8aba6 (the sso → oidc socket rename with the node-local migration that keeps already-wired edges revocable). The design text below is unchanged and still the argument of record; only the step table’s status column moved. Still open: step 3, step 5’s core/lifetime cascade (resolveEdgeLifetime stays pinned to external, so every destroy is closed by construction), and step 7.

Revision 3. The six echo-only cleanup-access socket jobs have been deleted, so the dead-job count is 28 → 22 and every remaining dead job undoes real state. The table below is a fresh measurement, not an edit of revision 2’s. One prerequisite is added that revision 2 did not know about: the integration table is never written on the install path, so step 6 cannot key teardown off a DB row (see The registry the rows would come from is unwired).

Revision 2. Rewritten after an adversarial review returned ship with changes. What moved: the dead-job count is remeasured across both catalog trees (26 → 28); the gating mechanism is now the core/lifetime scope parameter, not an operator flag and not an app-authored cleanup@boolean; the rollout reflects the renames that have already landed; and the project owner’s vApp constraint is recorded as a hard constraint rather than an open question.

Twenty-two job blocks in the catalog were written by app authors, are parsed by nothing, and have never executed. They are not stale leftovers; they are the teardown halves of integrations that ship today.

The catalog lives in two trees. ce/builtin/<app> is the source of truth for the four built-in apps (keycloak, openobserve, postgres, traefik); it is gitignored in this repo and staged into apps/ at build time. The first scan walked apps/ only and therefore missed it. Counts below are a YAML parse of every jobs.<phase> list in the CE repo’s builtin/ and this repo’s apps/, re-run at revision 3 rather than carried forward; a count nobody re-measures is a count that silently drifts, which is what happened to socket/discovery (revision 2 recorded 28 in apps/; a job has been added since).

surfacephasece/builtinappstotalstatus
*.plug.yamlunintegration01515dead
*.socket.yamlunintegration156dead
*.vsocket.yamlpost_deployment101dead
22
*.plug.yamlpost_deployment14243runs
*.socket.yamldiscovery22931runs
*.socket.yamlintegration22123runs
*.vsocket.yamlintegration21012runs
*.vsocket.yamlcleanup21012runs
all authored jobs11132143

2 of the 22 dead jobs live in the staged tree, which is why a scan of apps/ alone cannot see them:

  • ce/builtin/keycloak/sso.socket.yaml :: delete-oidc-client (unintegration)
  • ce/builtin/postgres/database.vsocket.yaml :: vapp-install-extensions (post_deployment)

The 15 dead plug jobs span 10 apps (defectdojo, fleet, gitea, immich×2, mailcow×2, mautic×2, nextcloud, sonarqube, windmill×2, wordpress×2). The 6 dead socket jobs span 6 apps (crowdsec, keycloak, minio, pmm, s3, wazuh).

The socket row fell from 12 to 6 because rollout step 0 deleted the six cleanup-access jobs (hockeypuck, immich, paperless, peertube, penpot, postiz) whose entire body was echo "<app> integration cleaned up". Every one of the remaining 21 unintegration jobs undoes state a real teardown has to undo: which is the property that makes waking the phase worth doing and deleting it unacceptable.

Lesson for the audit itself. Any catalog measurement that reads apps/ without staging ce/builtin is wrong by construction, and will stay wrong silently because the missing apps are the platform apps, the ones every other app plugs into. Every future catalog assertion must run against the staged tree.

The runners parse, exhaustively:

  • plugJobsDoc (plug_jobs.go:43-47): post_deployment ONLY.
  • SocketJobsDoc (socket_jobs.go:45-50): discovery, integration.
  • vsocketDoc (vapp_runner.go:495-507): discovery, integration, cleanup.

So cleanup executes on a vsocket and is silently ignored on a socket and a plug, and post_deployment is the mirror image: it executes on a plug and is silently ignored on a vsocket. The same word, in the same YAML dialect, in the same directory, with opposite behaviour, in both directions.

plug_jobs.go:41-42 tells the reader the other phases are “handled elsewhere”; they are not handled anywhere. integration_service.go:176-179 is honest about it (“the unintegration + cleanup job phases are out of scope here”) and IntegrationService.Disconnect (integration_service.go:180) has zero callers in the tree.

None of this fails loudly because yaml.Unmarshal into a typed struct discards unknown keys. An author writes a phase, the parser drops it, nothing logs.

The consequences are live, not theoretical

Section titled “The consequences are live, not theoretical”
  • Uninstalling a consumer never revokes its provider-side access. Ten apps wrote SSO/SMTP unwiring (mailcow, nextcloud, gitea, wordpress, …) that has never run once. A removed Nextcloud leaves its Keycloak client, its SMTP relay credential and its bucket IAM user live.
  • The only teardown that DOES run is the vApp path, and it runs unconditionally: UninstallVapp loops vsocket.Jobs.Cleanup with no gate of any kind (vapp_runner.go:437-445). Ten of the twelve vsocket cleanups destroy data: postgres/mariadb/mysql DROP DATABASE + DROP USER, mongodb dropDatabase, minio/s3 bucket rb --force, keycloak DELETE /realms/<realm>, redis SCAN+DEL over the whole namespace, frappe and nginx rm -rf. (The remaining two, wordpress and nextcloud uninstall-plugin, remove an app, not a dataset.)
  • The migration case that prompted the audit: a bucket holding 55.6 GiB of replicated Nextcloud data, adopted rather than provisioned, destroyed on teardown with no opt-out.

Of the dead jobs, three destroy data on the edge surface: apps/minio/s3.socket.yaml, apps/minio-classic/s3.socket.yaml and apps/s3/s3.socket.yaml, each rb --force. The rest release access, deregister an agent, unwire a config file, or, in the lone vsocket post_deployment case, install postgres extensions. That asymmetry is what makes waking the phases tractable: the blast radius of “parse unintegration” is three jobs wide, and all three are identified.

This count has already drifted once. It was written as “two, of 22” and became three when apps/minio-classic/s3.socket.yaml was added, a copy of apps/minio’s socket, destroy job included, with nothing in the tree that noticed. A hand-maintained inventory in a proposal is not a control, so the predicate is now enforced instead of counted: tests/unit/catalog/unintegration_destroy_gate_test.go walks every committed .plug.yaml / .socket.yaml, fails on ANY unintegration job that removes a bucket without lifetime: managed, and pins the inventory by name so the next provider cannot join silently. Treat the number in this paragraph as history and that test as the source of truth.

The persistence gap is the cause, not a coincidence

Section titled “The persistence gap is the cause, not a coincidence”

This is the single most important fact in the proposal, and it is why the dead phases cannot be fixed by adding a struct field.

VappState persists the entire rendered context (PlugData, VVars, Discovery, Preset/PresetResolvedVars (vapp_runner.go:70-110)) with the reason stated inline at :91-94:

“PlugData / VVars are persisted so the cleanup jobs can render against the same context the install used. Without this the {{ integration.plug.data.domain }} ref in the uninstall job has nothing to bind against and the cleanup runs blank.”

PersistedIntegration (install_runner.go:2925-2946) is five bare strings: Plug, ProviderStackName, ProviderTenant, ProviderEnvironment, ProviderConnection. No plug data, no socket data, nothing rendered.

The vApp surface persists its bind-time context and its teardown phase works. The integration-edge surface does not persist its bind-time context and its teardown phase does not exist. That is not two independent gaps; it is one gap and its consequence. A teardown job that says mc rb s3/$${PLUG_BUCKET} has nothing to render PLUG_BUCKET against.

Parsing the phase without fixing persistence is worse than leaving it dead: the jobs would run with blank variables. mc rb s3/ and DROP DATABASE IF EXISTS "" are not no-ops in every implementation, and rm -rf "$${DOC_ROOT}" with an empty DOC_ROOT is a catastrophe.

The registry the rows would come from is unwired

Section titled “The registry the rows would come from is unwired”

Revision 2 assumed the integration table was the natural place to hang teardown state; it already carries a connection_values JSON column, and DisconnectRemovedIntegrations (install_async.go) already deletes its rows on -i <ref>!. Neither assumption survives contact with the tree.

IntegrationService.Connect refuses any edge whose consumer plug and provider socket are not declared to the repository, and the ONLY writer of those declarations is IntegrationRepository.RegisterStack (integration_repository.go:134). Nothing outside _test.go calls it, and nothing outside the repository’s own zero-value initialiser constructs a PlugSpec. So on the install path the declarations are always empty, Connect can only return X404007PLGNOF, and no install ever writes an integration row. Three consequences follow, and step 6 must be designed around all three:

  • DisconnectRemovedIntegrations is a no-op in production. It looks like the natural hook for teardown; it is a walk over an empty table.
  • daemonRecoveryActions.WireIntegrations (daemon/services_recovery.go:312) is the one production caller of Connect, and it returns the first error; so a recover carrying edges fails on a lookup that cannot succeed. That is a separate bug, but it shares this root and should be fixed with it.
  • ListIntegrations already ships a state.json fallback for exactly this reason, with the cause named inline: “the thin install path records integration edges only in the consumer’s state.json; it never writes the deployment / integration rows the DB-backed registry reads”. The fallback is what makes shc integration ls look correct while the table is empty.

state.json is therefore the only surface that knows an edge exists, which is where PersistedEdgeState (step 4) has to live. It is also the surface ExecuteComposeDown already reads before it tears anything down (st.Integrations feeds revokeConnectionEdges), so step 6’s uninstall wiring needs no new source of truth, only the rendered context step 4 adds to it.

Why re-resolving at teardown is not a substitute for persisting

Section titled “Why re-resolving at teardown is not a substitute for persisting”

BuildIntegrationsContextForDeployment (integration_render.go:1505) rebuilds the whole integrations: namespace from state.json plus the deployment’s app/ copy, and export and backup both use it. It is the obvious shortcut, and it must not be taken on the teardown path:

  • It re-renders plug data against vars as they are NOW, not as they were at bind. That is the drift the Alternatives considered section already rejects.
  • For a connection-backed edge (-i <plug>=@<conn>) it routes through ActiveConnectionMinter.Mint (integration_render.go:691). Calling a cloud mint API is defensible on a render path; on an uninstall path it provisions a scoped credential moments before revokeConnectionEdges revokes one.
  • The mint-and-return channel is not replayable at all. Values a socket job emits as ::shc-connection-value::KEY=VALUE:: are folded into the render context in memory (MergeJobMintedValues) and never persisted, so any key that exists only because a job minted it renders empty on a re-resolve. No live unintegration job reads such a key today; apps/crowdsec is the one emitter in the catalog, and its own remove-bouncer reads plug data, but nothing in the tree stops the next one from doing so.

The project owner’s constraint, recorded verbatim:

vApps destroy by default, with no opt-out.

This is a hard constraint on the design, not a default to be revisited. Its consequences run through everything below:

  • The core/lifetime parameter described in §2 governs integration edges only. runVappJob never consults it.
  • A lifetime: key on a *.vsocket.yaml job is a validation error, not an ignored field. There must be no syntax that looks like a vApp opt-out.
  • The ten destructive vsocket cleanups stay unguarded, and that is now the documented intent rather than an oversight. Revision 1 proposed guarding them; that proposal is withdrawn.
  • Adoption of a pre-existing resource is therefore an edge operation, never a vApp. A vApp instance is its resource: postgres/database means “this database exists because this vApp exists”. If an operator needs to point at a bucket or database they already own and keep, that is -i <plug>=<provider> with lifetime: external, or -i <plug>=@<connection>. The 55.6 GiB bucket belongs on the edge path, and the fix for that incident is to make the edge path exist and be safe, not to weaken the vApp contract.

They are genuinely different and the distinction is worth preserving:

  • cleanup: this thing is being uninstalled. The resource is going away. Fires for a vApp instance and for an app.
  • unintegration: an edge is breaking, both endpoints survive. Fires on disconnect and on consumer uninstall.

Under that rule the catalog was misnamed in one direction only: the socket cleanup blocks (remove-bouncer, remove-agent, and the six cleanup-access jobs step 0 has since deleted) and the 15 plug cleanup blocks all fire when a CONSUMER goes away, edge teardown. They become unintegration. The 12 vsocket cleanup blocks are correct as-is. This rename has landed; see the rollout below.

The remaining misname is the one the recount exposed: post_deployment on a vsocket. ce/builtin/postgres/database.vsocket.yaml :: vapp-install-extensions means “after the vApp’s database exists”; post_deployment on a plug means “after the CONSUMER’s containers are up and healthy”. Same word, different event. The job is a second step of provisioning and belongs in jobs.integration after vapp-create-database, which already runs in declaration order. One file, one move.

And the parser must stop swallowing this class of bug. yaml.Unmarshal into a typed struct silently drops unknown keys, which is how 28 jobs (the 22 that remain plus the 6 step 0 deleted) got authored into the void. The fix is not Decoder.KnownFields(true); the job docs are partial views of files that also carry schema:, data:, services:, so strict decoding would reject valid catalog. The fix is to decode jobs into map[string]yaml.Node first and hard-error on any key outside the surface’s known phase set. Cheap, exhaustive, and it makes “authored but never executed” unrepresentable rather than merely tested-for.

2. core/lifetime: a scope parameter, not a flag and not an app boolean

Section titled “2. core/lifetime: a scope parameter, not a flag and not an app boolean”

Destruction is gated by one platform-owned parameter, lifetime, resolved through the existing SHC scope ladder in core/scope (core/scope/scope.go:24-38): Deployment → Stack → Environment → Tenant → Global. Hence core/lifetime: the core/ namespace marks it as platform-defined; the platform owns its name, its value domain, its default and its validation, so no app can misspell it and no app has to re-implement its truthiness.

Value domain: a closed enum, validated at parse time:

valuemeaning
externalThe resource pre-existed this edge or outlives it. Teardown releases access only. Default.
managedSHC provisioned this resource for this edge and may destroy it when the edge breaks.

Resolution, once, at bind time:

  1. explicit --lifetime managed on the install / connect invocation;
  2. core.lifetime through the scope cascade: shc config set core.lifetime managed --tenant dev makes a scratch tenant reclaim everything, while production keeps the global external;
  3. the plug’s declared default in <plug>.plug.yaml;
  4. external.

The resolved value is persisted on the edge (§3) and that persisted value is what teardown reads. Resolving at bind and freezing it is the same argument as persisting plug data: the disposition of a resource is a fact about how it came into existence, and a config change six months later must not silently convert “retain” into “destroy”.

How a job declares itself destructive. The author marks the job; the platform decides whether it runs:

jobs:
unintegration:
# Always runs: release the per-consumer credential.
- name: revoke-access
context: container
command: |
mc admin policy detach s3 "policy-$${PLUG_BUCKET}" --user "$${PLUG_ACCESS_KEY}" || true
mc admin user remove s3 "$${PLUG_ACCESS_KEY}" || true
# Runs only when the edge resolved to lifetime: managed.
- name: delete-bucket
lifetime: managed
context: container
command: mc rb s3/$${PLUG_BUCKET} --force

Two properties follow that the previous designs did not have:

  • Release-access and destroy-resource stop being the same job. The pre-split delete-s3-bucket did both in one command, so any guard that skipped the destroy also skipped the credential revocation, the exact leak the whole proposal exists to close. Splitting them is only possible because the gate is per-job. (Landed for all three s3 providers; the shape below is now what the catalogue ships, not an illustration.)
  • Shell truthiness leaves the blast radius. The runner compares an enum it parsed; no catalog job ever tests a boolean in sh again. This repo has been bitten by that class twice in the last week alone: the jinja fix at core/jinja/jinja.go:865-887 (a stringified "False" was truthy, so huly, affine and gitlab addressed a plaintext S3 endpoint over TLS) and its shell mirror, b9c50518 “a YAML bool never equals the string true in a shell test”. The in-flight cleanup@boolean guards already had to write case "$${PLUG_CLEANUP}" in true|True|TRUE|1) to survive it. A destructive guard is the last place that should depend on which of five spellings of true a template happened to emit.

Default is retain, and the failure modes are the argument. An orphaned bucket costs pennies and is swept later; a destroyed one costs whatever was in it. An edge whose persisted state predates this design, or was lost to a partial restore, resolves to external, so an unknown edge behaves like one that declined. That is what makes waking these phases safe rather than merely possible.

A KeepIntegrations operator flag was prototyped and has been reverted. It is worth being precise about why. The first reason is the one that would have shipped a data-loss bug; the other three apply to any flag-shaped fix, so a narrower flag in a better place would still have been the wrong answer.

  1. It was structurally in the wrong place. The gate sat inside ExecuteComposeDown, below the vApp early-return at install_runner.go:1440-1442:

    if (strings.Contains(opts.AppName, "/") && !IsExplicitInstallSource(opts.AppName)) ||
    strings.Contains(opts.StackName, "/") {
    return UninstallVapp(ctx, opts)
    }

    Every vApp uninstall returns there. The vApp path is the only path in the product that currently runs destructive teardown (vapp_runner.go:437-445). So the flag guarded exactly the code paths that had nothing to guard, and never once fired on the path that did, while reading, in the CLI help and in review, as though it protected everything. A flag that is silently inert on its most important case is worse than no flag: it converts an unguarded destroy into an unguarded destroy that someone has already signed off on.

  2. Per-invocation disposition is the wrong lifetime. A flag is a property of one command. Whether a bucket survives then depends on how the last uninstall happened to be typed. Worse, several teardown paths have no human at the keyboard at all: the lease reaper (crons_lease.go:199 → DefaultLeaseReapTrigger at :348), the clone/rollback seams (daemon/clone_adapters.go:525, 547, daemon/services_seams_clone.go:389, 404) and the post-teardown PurgeStackRow tail all reach uninstall with a synthesised request. A flag they cannot set is a flag that defaults them into whichever behaviour the code happened to pick, and the reaper, which removes deployments nobody is watching, is precisely the caller you least want guessing.

  3. It asked the wrong actor at the wrong time. The platform does not know whether a resource was created or adopted. Only the consumer’s operator knows, and they knew it at bind time, possibly months earlier, possibly a different person. Asking at teardown time is asking someone to recall a fact nobody wrote down.

  4. It left no trace. Nothing was persisted, so nothing could be audited, replayed, reported by shc ls, or reasoned about before running the uninstall. There was no way to answer “what happens if I remove this?” other than removing it.

core/lifetime inverts all four: it is resolved at bind (2, 3), by the party who knows (3), persisted on the edge (4), and consumed by the job runner rather than by the uninstall entry point, so there is no early-return it can sit below (1). Because it is a scope parameter rather than a flag, it also gains the tier resolution a flag structurally cannot have: an estate-wide posture, a per-tenant override for scratch environments, and a per-edge exception, all in one key.

Why not the app-authored cleanup@boolean convention

Section titled “Why not the app-authored cleanup@boolean convention”

Revision 1 proposed a per-plug cleanup@boolean schema field guarded in each job’s shell. It is strictly better than the flag and it is what the in-flight guards on apps/minio/s3.socket.yaml and apps/s3/s3.socket.yaml currently implement. It is still the wrong long-term mechanism:

  • It is re-implemented per app, in shell, against jinja’s stringified booleans: see the truthiness argument above. Every copy of that five-way case statement is another chance to write it wrong once, in the destroy path, in a job nobody runs until the day it matters.
  • It has no scope. There is no way to express “this whole tenant is scratch” or “this estate never destroys” without editing every plug.
  • It is invisible to the platform: shc cannot report it, validate it, or refuse a destroy, because it is just another opaque string in plug.data.
  • It conflates the two halves of teardown, as shown above.

The cleanup@boolean guards were not wasted work; they were the correct interim state (see rollout step 2), and they marked exactly which jobs got lifetime: managed. They now sit INSIDE those jobs as a second condition that can only add retention, so consumer intent and provenance both have to hold.

3. Persist the bind-time plug context; re-render socket data

Section titled “3. Persist the bind-time plug context; re-render socket data”

Give integration edges the same bind-time replay VappState already gives vApps: but not by symmetry alone. The two halves of the rendered context need opposite treatment, and the reason is a security boundary the vApp case does not have.

Plug data is replayed. Re-rendering it at teardown reads the consumer’s deployed app/ tree and its vars as they are at uninstall, not as they were at bind. If vars.postgres.database drifted between the two, the teardown job targets a different database than the one provisioned; it does not fail, it succeeds against the wrong resource. Bucket names, database names and usernames must come from the bind-time snapshot. This is the same contract port mappings and var overrides already have.

Socket data is re-rendered, deliberately. The provider’s admin credential is exactly the field you want current at teardown: a rotated superuser password must authenticate, and a replayed one will not. And an edge can be cross-tenant, where integration_render.go:507-520 already blanks every @secret socket field in the consumer-facing copy. Persisting provider secrets into the consumer’s state.json would hand the consumer’s tenant the very values that redaction exists to withhold. The provider is by definition still running when a consumer is being removed (if it is not, the teardown job could not execute anyway) so re-rendering from the live provider is both safe and required.

Where it goes. Not in PersistedIntegration. That struct is one of three that are converted positionally: IntegrationSpec (integration_render.go:50-72), PersistedIntegration (install_runner.go:2043-2064) and DiskIntegration (disk_lookup.go:297-310), each with a comment insisting ProviderConnection stay last because IntegrationSpec(pi) / PersistedIntegration(spec) / DiskIntegration(in) are direct conversions. Adding a map to one silently mis-maps the others, and IntegrationSpec is the type -i parses into; it has no business carrying a rendered blob.

Instead, add a sibling record modelled on VappState, stored as its own key in the deployment’s state.json (JSON, keyed by name, no positional coupling):

// PersistedEdgeState is to an integration edge what VappState is to a
// vapp instance: the bind-time context, replayed at teardown.
type PersistedEdgeState struct {
Plug string `json:"plug"`
Lifetime string `json:"lifetime"` // resolved core/lifetime, frozen at bind
PlugData map[string]any `json:"plug_data"`
}

The three positional structs are untouched. PersistedEdgeState inherits the same at-rest posture the vApp state file already has for the same reason (plug data carries consumer-side secrets such as clientSecret and the vApp DB password); nothing new is exposed, and the provider’s secrets are never written into it at all.

An edge with no PersistedEdgeState (pre-existing deployments, or a state file lost to a partial restore) renders integration.plug.data from the consumer’s deployed app/ tree as a fallback and is forced to lifetime: external. This is not a compatibility shim; it is the permanent rule for any edge whose bind-time context is unavailable. Unknown provenance never destroys.

runVappJob (vapp_runner.go:586-709), runOneSocketJob (socket_jobs.go:402-500) and runOnePlugJob (plug_jobs.go:143-211) are three retypings of one skeleton: default context → validate → render → $$→$ unescape → renderHookEnv → dispatch. The $$ unescape alone is duplicated verbatim at vapp_runner.go:668, socket_jobs.go:423, plug_jobs.go:164.

They converge only in the last four inches: ExecuteResolvedJob, renderHookEnv, ResolveVaultAutogenInVars, LoadVars, RenderMapStrings. There is no shared “build a job template context” function, and the drift shows:

  • Different defaults for an omitted context:: vapp service (vapp_runner.go:597-600), socket container (socket_jobs.go:409-411), plug service (plug_jobs.go:150-153). Unifying naively changes the meaning of every catalog job that omits the key.
  • Robustness on one side only. Socket-only: context: http probes (socket_jobs.go:360), max_retries / retry_interval (socket_jobs.go:81-87, 298-301), dockerNetworkAvailable fail-fast with X500963NETUNR (socket_jobs.go:473-474), SocketServiceHost fallback (socket_jobs.go:484), mint-and-return via ::shc-connection-value:: (socket_jobs.go:239), @secret redaction on cross-tenant edges (integration_render.go:507-520). The vapp runner rejects http outright (vapp_runner.go:602) and has no retry concept at all, which is why a vApp teardown against a half-stopped parent just loses.
  • discovery means opposite things. Socket discovery is a readiness probe whose stdout is DISCARDED (socket_jobs.go:307). Vapp discovery is value-producing: stdout is parsed, published as {{ discovery.<job> }}, and it triggers a SECOND render of the vplug data: block (vapp_runner.go:302, 329).
  • pickJobContainerNetwork is defined in vapp_runner.go:730 and borrowed by both the socket runner (socket_jobs.go:458) and the render path (integration_render.go:814), while socket_jobs.go:10 claims it “mirrors vapp_runner.go::runVappJob”. The dependency runs backwards from the documentation, and the helper hardcodes DockerProjectName("default", …) (vapp_runner.go:757), a single-tenant vApp assumption sitting in the cross-tenant path, with the comment at :752-756 admitting it.

Target: one job runner and one context builder, parameterised by lifecycle event and provider identity. cleanup vs unintegration becomes two values, not two code paths, and lifetime is checked in exactly one place.

5. The trap that decides the whole refactor

Section titled “5. The trap that decides the whole refactor”

vars resolves four ways, two of them ninety lines apart in one file:

surfacevars iswinneranchor
vplug data:vvars (+ --var)the instancevapp_runner.go:289
vsocket data:SocketVars(parentVars, vvars); vvars fill gaps, parent overwritesthe parentvapp_runner.go:264, fn at :542-551
vapp job commandparentVars merged under vvarsthe instancevapp_runner.go:630-641, bound at :649
socket / plug jobprovider vars for the socket render, consumer vars for the plug render; never mergedn/aintegration_render.go:359-364 / :461-466

So inside ONE .vsocket.yaml, {{ vars.password }} is the parent superuser in data: and the per-vapp user in jobs.*[].command. This inversion is deliberate and load-bearing. ce/builtin/postgres/database.vsocket.yaml:17 publishes adminPassword: "{{ vars.password }}", and ce/builtin/postgres/database.vvars.yaml declares its own password: ref+vault://vapp_db_password?autogen, the per-vapp DB user’s secret. The comment at vapp_runner.go:534-541 spells out why parent must win there: a vapp’s vvars password must not shadow the parent superuser password the socket publishes. Invert it and every vApp provisions its database with the wrong credential, succeeding, silently, until something tries to log in.

Three consequences, all of which bind work already in flight:

  1. A shared context builder that picks one merge order silently mis-provisions credentials. Any unification must keep merge order a per-surface property, with a table test pinning all four before a line of the collapse is written.
  2. The parent-wins order is pinned in two places: the install render (vapp_runner.go:264) and the teardown replay (vapp_runner.go:429). Both call SocketVars. A refactor that fixes one and not the other makes install and uninstall disagree about which database they are talking about.
  3. The in-flight vars. → vvars. rename sweep (17 refs) must not be applied blanket. In postgres and s3, vars.username / vars.password inside a vsocket data: block mean the parent’s admin credentials by design; renaming them to vvars. swaps admin for instance and inverts exactly the case the comment warns about.

Ordered so each step is separately revertable and none silently destroys data. Status reflects the working trees as of this revision.

#stepstatus
0Delete the six echo-only unintegration jobs (cleanup-access in hockeypuck, immich, paperless, peertube, penpot, postiz). Their whole body was an echo; keeping them means step 6 must reason about jobs that undo nothing, and every count in this proposal overstates the work. The 21 that undo real state stay, as does unintegration in modules/job/service.go’s phaseList; the inventory entry is what keeps shc job ls honest about jobs that exist but do not run.DONE (uncommitted), 6 files under apps/, 42 lines, no other reference in the tree
1Rename the 23 misnamed blocks (8 socket + 15 plug cleanup → unintegration). Catalog-only, still inert. Clean break, no alias.DONE (uncommitted), 23 catalog files renamed under apps/, plus one stale doc-comment fix in apps/crowdsec/wazuh.plug.yaml. apps/minio, apps/pmm, apps/s3 and ce/builtin/keycloak already shipped unintegration, so nothing to rename there.
2Guard the destructive edge jobs. The interim cleanup@boolean convention on the rb --force jobs.DONE, and superseded by step 5’s catalogue half. Landed at THREE files, not two; apps/minio-classic/s3.socket.yaml arrived after this row was written. The cleanup allow-list survives inside each destroy job as a second, retain-only condition; it is no longer the gate.
3Move vapp-install-extensions from jobs.post_deployment to jobs.integration in ce/builtin/postgres/database.vsocket.yaml, and hard-error on unknown jobs.<phase> keys in all three parsers.TODO, one catalog line, one parser change; closes the class.
4Add PersistedEdgeState (plug data + resolved lifetime) and write it at bind. No behaviour change; new deployments start carrying it. Edges without it resolve to external.DONE, ce 988d4c2 (modules/app/edge_state.go, written into state.json’s edge_states by every successful deploy). Identifiers freeze rendered; @secret fields are held as their RAW templates and re-rendered at teardown; socket data is never persisted (cross-tenant redaction). Mutation-audited: forcing BuildPersistedEdgeStates to nil reddens the install-persists test by name; forcing the re-derive path reddens seven tests with “revoked the WRONG resource, successfully”.
5Land core/lifetime: the scope key, the --lifetime flag, the per-job lifetime: marker and its runner gate, and the vsocket validation error that forbids the key there. Convert the step-2 cleanup@boolean guards to lifetime: managed jobs, splitting release-access from destroy-resource.PARTIALLY DONE. The per-job marker + runner gate are in ce (modules/app/unintegration_runner.go, X400074UNILFT / X200925UNISKP), and the catalogue conversion has landed: all three s3 sockets now ship delete-bucket with lifetime: managed, and minio/minio-classic ship an ungated revoke-bucket-access beside it (apps/s3 mints no credential, so it has no revoke half). Still TODO: the core/lifetime scope key, the --lifetime flag, resolveEdgeLifetime (still pinned to external, so every destroy is skipped by construction), and the vsocket validation error forbidding lifetime: there.
6Parse and run unintegration on socket and plug, from the uninstall path while containers are still up, and from Disconnect (which gets its first caller).DONE, ce 63e44b6 (uninstall leg: RunUnintegrationJobs sited after pre_uninstall, before the compose verb, in ExecuteComposeDown) + 45aae15 (disconnect leg: SnapshotDisconnectedEdges before ExecuteCompose, DisconnectedEdges.Run after it succeeds, before the DB row drop). Plug jobs before socket jobs, per-job authored timeouts, pipeline/steps: shapes refused loudly (X500089UNISHP). A failed job never fails the teardown: it lands on the retained-cleanup channel as X500927UNILEK and the uninstall completes.
7Collapse the three runners, with the four vars merge orders pinned by tests first.TODO

Steps 3–6 are small and independently useful. Step 7 is the real refactor and should not be bundled with them.

Ordering constraints that are not negotiable:

  • 5 before 6. No phase is parsed until the gate that governs it exists.
  • 4 before 6. No teardown job runs until there is a bind-time context to render it against, and a rule for what happens when there isn’t.
  • 4 lands in state.json, not the integration table. The table has no install-path writer (see The registry the rows would come from is unwired), so an edge state hung off a row would be written for no edge that exists.
  • 6 runs before composeDown, not after. Nine of the 21 live jobs are context: service against the CONSUMER’s own container (nextcloud’s occ, gitea’s admin auth delete, mautic’s console cache:clear), and three more are context: container jobs that curl the consumer over the stack network (immich×2, sonarqube). The only point in ExecuteComposeDown where the consumer is still up is the pre_uninstall hook site; anywhere below the compose verb, each of those twelve silently takes the “could not authenticate, skipping” branch its own author wrote, and exits 0.
  • 7 last. Collapsing runners while the surfaces still disagree about vars is how the credential inversion ships.
riskmitigation
Waking the jobs starts destroying dataOnly three destroy anything, and all three now carry lifetime: managed, converted BEFORE the phase is parsed (step 5 before 6), with resolveEdgeLifetime still pinned to external so the destroy is closed by construction. tests/unit/catalog/unintegration_destroy_gate_test.go keeps the claim true as the corpus grows.
A woken job runs but silently does nothing, and the runner reports successThree of the 21 live unintegration jobs use shapes NEITHER existing runner parses: context: pipeline + steps: (apps/fleet/smtp.plug.yaml) and volumes: (apps/wordpress/{smtp,sso}.plug.yaml). Dropping volumes: is the dangerous one: wp core is-installed against an unmounted /var/www/html exits non-zero and the job’s own guard then exit 0s, so a no-op is indistinguishable from a clean teardown. Step 6 asserts the RESOLVED ResolvedJobSpec per surviving job, not just that the phase was reached
Teardown jobs run with blank vars on old edgesStep 4 first; an edge with no PersistedEdgeState falls back to the deployed app/ tree AND is forced to external, so a blank render can never reach a destroy
core/lifetime resolves differently at teardown than at bindResolved once at bind and frozen in PersistedEdgeState; teardown reads the persisted value, never the live cascade
A lifetime: key gets added to a vsocket job, creating a de-facto vApp opt-outValidation error, not an ignored field; the owner’s constraint is enforced by the parser, not by convention
A failing teardown job blocks uninstallNever abort; log per job with X500009INTJOB and continue; the deployment must still be removable when the provider is half-down. Matches the existing best-effort disposition at vapp_runner.go:440-443
Silent no-op looks like successLoud on failure, and loud on skip: "lifetime=external, retaining bucket <name>", with the resolved lifetime reported before the uninstall runs
Release-access is skipped along with destroySplit into separate jobs (landed); only the destroy job carries lifetime:, and a test asserts the revoke sibling never does
Unified runner flips an omitted context: defaultPin the three current defaults per surface in tests before collapsing
Unified context builder picks one vars merge orderTest all four orders first; keep merge order a per-surface property; check both SocketVars call sites
A future catalog scan misses ce/builtin againEvery catalog assertion runs against the staged tree, and CI fails if the assertion runs against an unstaged apps/
  • Parser exhaustiveness, as an error not a test. Every jobs.<phase> key outside a surface’s known set hard-fails at load. The accompanying test asserts the catalog parses clean; run against the staged tree, so ce/builtin is included by construction.
  • Table test pinning the four vars merge orders, including the vsocket data: vs job-command inversion, with the postgres/database admin-vs-instance password case named explicitly, and both SocketVars call sites (install render and teardown replay) covered.
  • Lifetime resolution: each tier of the cascade wins over the next; the --lifetime flag beats the cascade; the plug default beats the built-in; absent everything yields external; an unknown value is a parse error, not a fallback.
  • Lifetime freezing: change core.lifetime after bind, tear down, assert the bind-time value governed.
  • vApp constraint: a vsocket job carrying lifetime: fails validation; a vApp uninstall runs its destructive cleanup with core.lifetime=external set at every tier.
  • Teardown behaviour: runs on consumer uninstall while containers are up; releases access unconditionally; destroys only at lifetime: managed; continues past a failing job; forces external and says so when PersistedEdgeState is absent.
  • Round-trip: install → uninstall with lifetime: managed destroys; with external the resource survives and a reinstall re-adopts it, with the same bucket/database name the first bind used.
  • Disconnect gets its first test alongside its first caller.

Parse unintegration only, leave the vsocket post_deployment dead. Fixes 21 of 22 and leaves one word still meaning two things on two surfaces. Rejected: the one remaining case is a single catalog line.

Make cleanup the single name everywhere. Fewer renames (12 vsocket files already use it), but it collapses two genuinely different lifecycle events (“I am being removed” and “an edge to me broke”) into one word, and a provider must distinguish them: remove-bouncer unwires one consumer, delete-realm destroys everything.

KeepIntegrations operator flag. Prototyped and reverted. Four reasons, detailed above; the decisive one is that its gate sat below the vApp early-return at install_runner.go:1440-1442, so it never fired on the only path that currently destroys anything.

App-authored cleanup@boolean in plug data. Better than the flag, and the correct interim state, but it re-implements a destroy gate in shell once per destructive job against jinja’s stringified booleans, has no scope, is invisible to the platform, and cannot split release-access from destroy-resource. Superseded by core/lifetime; the in-flight guards convert.

Guard the vsocket destroyers too. Revision 1 proposed this. Withdrawn; it contradicts the owner’s constraint that vApps destroy by default with no opt-out.

Re-render plug data at teardown instead of persisting it. Cheaper, and it is what a narrow fix does, but it binds teardown to vars as they are at uninstall rather than at bind, and silently targets the wrong resource after var drift. Retained only as the forced-external fallback for edges with no persisted state.

Persist socket data alongside plug data, mirroring VappState wholesale. Rejected on two counts: a rotated provider credential must be current at teardown or the job cannot authenticate, and on a cross-tenant edge it would write the provider secrets that integration_render.go:507-520 deliberately redacts into the consumer’s own state file.

pre_uninstall / post_uninstall app hooks. Both are fully wired (types.go:97-98, hooks.go:29-30, install_runner.go:1471, 1550, service.go:654, 659) and used by zero apps in either tree. Revision 1 proposed folding them into an app-level cleanup. Deferred: they carry a real before/after-teardown ordering distinction that cleanup does not, they are not part of the integration surface, and folding two working-but-unused hooks is churn that would enlarge an already-large change. Revisit once the first app needs one.