Skip to content
SHC Docs

The mailbox reconciler: the identity-broker that makes mail zero-touch

Status: open design. The package is built at apps/standalone/mailbox-reconciler and has never run against a live estate. The §3 ruling is made (broker rulings, 2026-08-14).

As written: BUILT, UNPROVEN. The package is authored at apps/email/mailbox-reconciler/. Its 44 tests are green. It has never run against a live mailcow or a live Keycloak, and it must not be pointed at an estate before the probe plan in §5 is executed. One owner ruling is owed (§3) and one platform gap is named (§4).

This is the component docs/proposals/2026-08-08-identity-and-edge-brokers.md §1 called for, built. That proposal established the doctrine (sockets mint per-consumer at deploy time, brokers act per-user at runtime; edge-brokers run consumer-side, identity-brokers are provider-adjacent) and left this one “NOT built… needs an owner ruling on its destructive edge”. The ruling is still owed. What changed is that the safe half now exists and the destructive half is shipped OFF, so the ruling can be made against a real artifact rather than a sketch.


One container (alpine/curl, already pinned by two other packages in this catalogue), one POSIX sh loop, no jq, no apk add, no new language runtime. Every pollInterval seconds, 300 by default:

  1. Preconditions. Mailcow URL/host/domain, the sealed admin key, the Keycloak host and admin credentials, the realm. Any one missing → one log line naming which one, and the pass ends having written nothing.
  2. The authsource gate. GET /api/v1/get/identity-provider must report "authsource":"keycloak", or no mailbox is created. §2 explains why this gate is the difference between a working broker and one that mass-produces mailboxes nobody can log into.
  3. Desired set. A master-realm admin token, then either GET /admin/realms/{realm}/users?briefRepresentation=true&enabled=true or, with groupPath set, GET /admin/realms/{realm}/group-by-path/{path} → GET /admin/realms/{realm}/groups/{id}/members. Paginated at 100. Addresses derive from the user’s realm email (default; users off the mail domain are skipped, nothing is invented) or from <username>@mailDomain.
  4. Actual set. GET /api/v1/get/mailbox/all/{domain}.
  5. Create the difference with POST /api/v1/add/mailbox, carrying authsource: keycloak. Success is decided by a read-back, never by the POST’s response; mailcow answers HTTP 200 with a "danger" body for payloads it rejected outright. A verified create appends a ledger row.
  6. Self-heal. A ledgered mailbox that no longer exists in mailcow has its row dropped.
  7. Disappearance. Ledgered addresses no longer in the desired set have an absence counter incremented and deletionPolicy applied.

Level-triggered. Every pass recomputes both sets from scratch. Nothing is remembered about events, so a missed pass costs latency and nothing else and a restart is not a special case. The ledger records only what this loop created, which is the property that makes deletion expressible without ever being able to reach a mailbox somebody else made.

The mint-and-return constraint, and what it costs operationally

Section titled “The mint-and-return constraint, and what it costs operationally”

mailcow’s admin key is granted per-consumer over the mint-and-return channel: the socket publishes apiKey: "" and emits the real value from its grant-admin-api job. ce folds that into the integrations namespace with MergeJobMintedValues, and exactly one consumer render reads that namespace: post_install / post_update. Not compose (rendered before the grant job runs, on every deploy, forever), and not a plug post_deployment job (renders the static socket data, where the value is the empty string).

So the package ships a hook job that seals the key (::shc-hook:secret:mailcow.apiKey=…:: → vault → ref+vault:// ref in the deployed vars and the manifest’s var_overrides → the container env from the next deploy on) and primes it into the state volume at 0600 so the already-running container converges on install day. The seal is what makes it durable; the primed copy is what makes it immediate.

The seal is mint-once (vault.SeedHookSecretViaVault is read-back-before-mint), so a rotated mailcow key never replaces it. The symptom is failing passes against a dead credential, never silent drift; the escape is shc update <stack> --var mailcow.apiKey!.


2. The three upstream facts this rests on, with citations

Section titled “2. The three upstream facts this rests on, with citations”

Every claim below was read out of source, not remembered.

(a) add/mailbox accepts authsource, but only when mailcow’s own IdP config already names that source. From data/web/inc/functions.mailbox.inc.php (fetched 2026-08-09), the add → mailbox case initialises $authsource = 'mailcow' and overrides it only under:

if ($_data['authsource'] == "mailcow" ||
in_array($_data['authsource'], array('keycloak','generic-oidc','ldap'))
&& $iam_settings['authsource'] == $_data['authsource']){
$authsource = $_data['authsource'];
}

and the column is written by INSERT INTO mailbox (…, authsource, active) VALUES (…, :authsource, :active).

Consequence, and it is the whole reason for the gate in step 2: with no oidc plug on the mailcow stack, a mailbox created by this broker silently lands authsource='mailcow' and is then refused at SSO login by identity_provider('verify-sso') with “The user’s authentication source is not of type OIDC” (apps/email/mailcow/README.md documents that refusal and the one-row UPDATE repair mailcow has no UI for). An ungated broker would produce that state in bulk. So the loop reads the IdP config first and holds off on creating, naming the remedy.

(b) A non-mailcow authsource clears the password. Same file:

if ($authsource != 'mailcow'){
$password = ''; $password2 = ''; $password_hashed = '';
}

The loop still sends a random 32-char password; it costs nothing, it keeps the create valid if the IdP config is torn down between the gate and the POST, and the value is discarded by both sides. Mail clients authenticate with app passwords, minted per consumer by whoever holds the mailcow grant; this broker mints none.

(c) The Keycloak endpoints exist and behave as assumed. GET /admin/realms/{realm}/group-by-path/{path} is @GET @Path("group-by-path/{path: .*}") public GroupRepresentation getGroupByPath(...) in RealmAdminResource.java. GET .../groups/{id}/members takes first/max/briefRepresentation in GroupResource.java, and returns direct members only, so a parent group path will not pick up subgroups (documented in the package README rather than worked around). GET .../users takes enabled and briefRepresentation among its query params (UsersResource.java), and brief representation is id/username/createdTimestamp/firstName/lastName/email/enabled/emailVerified/ federationLink (ModelToRepresentation.java), flat except for the temp-admin attribute map, which is exactly the nesting the brace-depth splitter exists to survive.

The mailcow routes (get/mailbox/all/{domain}, add/mailbox, edit/mailbox with {"items":[…],"attr":{…}}, delete/mailbox with a bare array) are all already exercised by shipped jobs in apps/email/mailcow/, so those are in-repo precedent rather than new claims.


What happens to a mailbox when its Keycloak user disappears. The package ships deletionPolicy: retain and does not need a ruling to be useful; the creation half is the zero-touch half, and a retain loop is a working loop. The ruling decides whether an estate may ever set anything else.

valueeffectreversible
retain (shipped default)a log line; no write of any kindn/a
disableedit/mailbox {"active":"0"}; mail kept, login and delivery stop; re-enabled if the user returnsyes
deletedelete/mailbox; mailbox and mail gone, app_passwd cascadesno; only a restic restore of the mailcow stack

The argument for retain as the default, restated so the ruling is made against it: a Keycloak user “disappearing” is frequently not a departure. An admin editing a group, a flapping user-federation link, a realm export/import, a mistyped groupPath, or an addressSource that filters everyone out all make users vanish from a listing. Under retain each of those is a log line. Under delete some of them are data loss, and this estate has no automated backup on either production estate (memory: no_backup_schedules_on_either_estate), so “restore the mailcow stack” is not currently a real recovery path.

Four properties hold under every policy, and they are what would make a delete ruling survivable if it is made:

  • Only ledgered addresses are eligible. A mailbox this loop did not create is never touched. Pre-existing mailboxes are skipped at creation and never adopted into the ledger, so they can never become eligible.
  • Grace window. graceCycles (3) consecutive passes of continuous absence. The counter resets the instant the user reappears and survives restarts.
  • Mass-absence guard. An empty desired set beside a non-empty ledger refuses the destructive half outright and says why.
  • dryRun reports every action, creates included, and performs none.

The ruling should also settle the cascade question. Deleting a mailbox cascades app_passwd; disabling it does not. If a per-user broker (Nextcloud Mail’s bridge, say) holds credentials for a departed user, disable leaves those credentials live against a mailbox that no longer authenticates its owner. Whether the reconciler should also invoke mailcow-revoke-app-passwords for the departed address, or whether that stays an operator action, is part of the same decision.

An agent must not flip this var. It is not a step in getting the loop working, and the package’s tests execute the dispatch block once per policy value specifically so that “make it pass” can never mean “make it delete”.


(a) apps/VERSION_IMAGE_DIVERGENCE needs one line, or tests/unit/catalog/app_version_tracks_image_test.go reds; it requires app.yaml’s version to be a prefix of some rendered image’s tag, and "1.0" is not a prefix of "8.21.0":

mailbox-reconciler untracked adapter contract version; the only image is the alpine/curl the reconcile loop runs in

Exactly the exit external-imap and nine other adapters already take.

(b) A keycloakAdmin socket on ce’s keycloak. There is none today, measured on this tree: 45 distinct *.socket.yaml stems, none of them that. apps/security/midpoint says so in its own words (compose.yaml:715, README.md:59) and ships no plug against it, and its tests assert the absence (tests/tests.bats:355-358, 395). So this package binds the socket that exists, oidc, and pays two prices:

  1. It holds master-realm admin credentials to do three read-only things (view-users, query-users, query-groups). adminUser/adminPassword are ordinary published socket data on oidc.socket.yaml, so any consumer of that socket already holds the whole Keycloak; this package does not widen the exposure, it inherits it.
  2. It mints an OIDC client it never uses. create-oidc-client belongs to the socket and runs for every plug bound to it; there is no way to plug this socket and not mint a client. It is minted in the least dangerous shape the socket can produce: publicClient: true (no stored secret; the socket then also sets directAccessGrantsEnabled:false) with one absolute, non-routable callback (http://127.0.0.1:1/…; the redirect guard refuses an undeclared field and a bare *, so some URI must be named).

The socket that would fix both: mint a per-consumer service-account client and grant it a NAMED subset of realm-management roles. That replaces a whole-Keycloak credential with a revocable, realm-scoped, read-only one, and turns the collateral client into the credential. midpoint waits on the same socket for the same reason; that makes two consumers, which is the usual bar for building it.

(c) x-shc-scale: max: 1 is declarative. The singleton constraint is real (two instances would race the same create, and the ledger is a plain file with no locking) and the engine that would enforce it is not built (memory: speculative_features_not_in_python). Until it is, “do not scale this service” is documentation.


5. THE LIVE-PROBE PLAN: what must be proven before this runs near an estate

Section titled “5. THE LIVE-PROBE PLAN: what must be proven before this runs near an estate”

Nothing below has been done. Run it on a throwaway mailcow + Keycloak pair, never on bitspur or ffx, and never with deletionPolicy set to anything but retain until P6.

Preconditions for every probe: the mailcow stack has its own oidc plug attached (probe P0 asserts it), and the reconciler is installed with --var dryRun=true.

#ProbePasses when
P0Install with both integrations, dryRun=true. Read docker logs for one full pass.The pass line names the right realm, group, domain, and a desired= count that matches the group’s membership. No precondition warning.
P1The seal. Inspect the deployed vars.yaml and the manifest var_overrides after install; then shc update and docker inspect the container env.mailcow.apiKey is a ref+vault:// ref after install, and the container env carries the plaintext after the update. The state volume’s api-key is 0600 and matches.
P2The authsource gate. Detach the mailcow stack’s oidc plug and let one pass run with dryRun=false.ZERO mailboxes created; the log names the remedy. Re-attach, and the next pass creates.
P3★ THE LOAD-BEARING ONE: authsource lands and SSO works. With dryRun=false, let it create one mailbox for a test realm user. Read the row back two ways: GET /api/v1/get/mailbox/<addr> and SELECT authsource FROM mailbox WHERE username='<addr>' in mysql-mailcow. Then log into the mailcow UI as that user via the SSO button.The DB says keycloak and the SSO login succeeds. If the API projection omits authsource, record that; the package’s post-create warning is best-effort and this probe is the only thing that can measure it.
P4Idempotence + scale. Let three consecutive passes run over a realm of >200 users (exercises the 100-per-page paginator on a real server).Pass 1 creates N, passes 2 and 3 create 0 and write nothing. The paginator terminates; desired= equals the realm count.
P5retain is inert. Remove a test user from the realm/group. Let 5 passes run at the default policy.The log reports the absence with a rising counter; the mailbox is untouched (active=1, mail intact, app passwords still authenticate). Zero POSTs in mailcow’s own access log for that address.
P6disable, then return. Only after the §3 ruling permits it. Set deletionPolicy=disable, graceCycles=2. Remove the user; let 3 passes run. Then put the user back; let 1 pass run.The mailbox goes inactive after pass 2 (not 1), the "active" read-back matches one of the shapes the loop accepts, mail is still on disk, and the return re-activates it. If the read-back shape does not match, the loop will log “did not read back inactive” forever; that is the safe failure, and this probe is what finds it.
P7delete. Only if the ruling permits it, and only on a mailbox with a known message in it.The mailbox is gone, its app_passwd rows cascaded, and the ledger row is dropped. Confirm from the mailcow DB, not from the API alone.
P8The mass-absence guard. Point groupPath at a group that does not exist, then at an empty one.Missing group → the pass aborts naming the path, nothing changes. Empty group with a non-empty ledger → the GUARD lines fire and the destructive half is skipped under every policy.
P9Credential failure modes. Rotate the mailcow key on the provider without re-sealing; separately, wrong Keycloak admin password.Both surface as failing passes with a named cause, the heartbeat stops, and the container goes unhealthy, never a silent no-op and never a crash loop.
P10Restart + volume loss. Restart the container mid-window; then reinstall with the volume removed.Restart: counters and ledger survive. Volume loss: the ledger is empty, so the disappearance half stops working and the create half re-converges; it must never delete anything after a volume loss.

Stop conditions. Any probe that writes to a mailbox the loop did not create is a hard stop. Any probe where retain produces a write is a hard stop.


6. Two bugs the offline harness already caught

Section titled “6. Two bugs the offline harness already caught”

Recorded because they are the shape of what the live probes are for, and because both were silent: a wrong answer, never an error.

  1. A create was un-ledgered in the same pass. actual is snapshotted before the creates, and the self-heal drops any ledgered address missing from actual; every row a pass wrote was therefore deleted again immediately. The ledger stayed permanently empty, so the disappearance half could never act under any policy. Fixed by folding verified creates back into actual before the self-heal; pinned by a test that asserts the ordering.
  2. The JSON extractors assumed space-free JSON. Both servers emit compact JSON today, and every other job in this catalogue assumes it, but the failure mode is not an error, it is an empty desired set, i.e. this broker quietly provisioning nobody while reporting a clean pass. Now whitespace-tolerant, and a pass that sees users but derives no addresses warns and names both likely causes.

Both were found by extracting the rendered command: and running real reconcile() passes against a stubbed curl. Neither static review nor shc validate saw either one.