The mailbox reconciler: the identity-broker that makes mail zero-touch
Status: open design. The package is built at
apps/standalone/mailbox-reconcilerand has never run against a live estate. The §3 ruling is made (broker rulings, 2026-08-14).
As written: BUILT, UNPROVEN. The package is authored at
apps/email/mailbox-reconciler/. Its 44 tests are green. It has never run
against a live mailcow or a live Keycloak, and it must not be pointed at an
estate before the probe plan in §5 is executed. One owner ruling is owed (§3)
and one platform gap is named (§4).
This is the component docs/proposals/2026-08-08-identity-and-edge-brokers.md
§1 called for, built. That proposal established the doctrine (sockets mint
per-consumer at deploy time, brokers act per-user at runtime; edge-brokers run
consumer-side, identity-brokers are provider-adjacent) and left this one
“NOT built… needs an owner ruling on its destructive edge”. The ruling is still
owed. What changed is that the safe half now exists and the destructive half is
shipped OFF, so the ruling can be made against a real artifact rather than a
sketch.
1. What it does
Section titled “1. What it does”One container (alpine/curl, already pinned by two other packages in this
catalogue), one POSIX sh loop, no jq, no apk add, no new language runtime.
Every pollInterval seconds, 300 by default:
- Preconditions. Mailcow URL/host/domain, the sealed admin key, the Keycloak host and admin credentials, the realm. Any one missing → one log line naming which one, and the pass ends having written nothing.
- The authsource gate.
GET /api/v1/get/identity-providermust report"authsource":"keycloak", or no mailbox is created. §2 explains why this gate is the difference between a working broker and one that mass-produces mailboxes nobody can log into. - Desired set. A master-realm admin token, then either
GET /admin/realms/{realm}/users?briefRepresentation=true&enabled=trueor, withgroupPathset,GET /admin/realms/{realm}/group-by-path/{path}→GET /admin/realms/{realm}/groups/{id}/members. Paginated at 100. Addresses derive from the user’s realm email (default; users off the mail domain are skipped, nothing is invented) or from<username>@mailDomain. - Actual set.
GET /api/v1/get/mailbox/all/{domain}. - Create the difference with
POST /api/v1/add/mailbox, carryingauthsource: keycloak. Success is decided by a read-back, never by the POST’s response; mailcow answers HTTP 200 with a"danger"body for payloads it rejected outright. A verified create appends a ledger row. - Self-heal. A ledgered mailbox that no longer exists in mailcow has its row dropped.
- Disappearance. Ledgered addresses no longer in the desired set have an
absence counter incremented and
deletionPolicyapplied.
Level-triggered. Every pass recomputes both sets from scratch. Nothing is remembered about events, so a missed pass costs latency and nothing else and a restart is not a special case. The ledger records only what this loop created, which is the property that makes deletion expressible without ever being able to reach a mailbox somebody else made.
The mint-and-return constraint, and what it costs operationally
Section titled “The mint-and-return constraint, and what it costs operationally”mailcow’s admin key is granted per-consumer over the mint-and-return channel:
the socket publishes apiKey: "" and emits the real value from its
grant-admin-api job. ce folds that into the integrations namespace with
MergeJobMintedValues, and exactly one consumer render reads that namespace:
post_install / post_update. Not compose (rendered before the grant job
runs, on every deploy, forever), and not a plug post_deployment job (renders
the static socket data, where the value is the empty string).
So the package ships a hook job that seals the key
(::shc-hook:secret:mailcow.apiKey=…:: → vault → ref+vault:// ref in the
deployed vars and the manifest’s var_overrides → the container env from the
next deploy on) and primes it into the state volume at 0600 so the
already-running container converges on install day. The seal is what makes it
durable; the primed copy is what makes it immediate.
The seal is mint-once (vault.SeedHookSecretViaVault is
read-back-before-mint), so a rotated mailcow key never replaces it. The symptom
is failing passes against a dead credential, never silent drift; the escape is
shc update <stack> --var mailcow.apiKey!.
2. The three upstream facts this rests on, with citations
Section titled “2. The three upstream facts this rests on, with citations”Every claim below was read out of source, not remembered.
(a) add/mailbox accepts authsource, but only when mailcow’s own IdP
config already names that source. From
data/web/inc/functions.mailbox.inc.php (fetched 2026-08-09), the add →
mailbox case initialises $authsource = 'mailcow' and overrides it only
under:
if ($_data['authsource'] == "mailcow" || in_array($_data['authsource'], array('keycloak','generic-oidc','ldap')) && $iam_settings['authsource'] == $_data['authsource']){ $authsource = $_data['authsource'];}and the column is written by
INSERT INTO mailbox (…, authsource, active) VALUES (…, :authsource, :active).
Consequence, and it is the whole reason for the gate in step 2: with no
oidc plug on the mailcow stack, a mailbox created by this broker silently
lands authsource='mailcow' and is then refused at SSO login by
identity_provider('verify-sso') with “The user’s authentication source is
not of type OIDC” (apps/email/mailcow/README.md documents that refusal and
the one-row UPDATE repair mailcow has no UI for). An ungated broker would
produce that state in bulk. So the loop reads the IdP config first and holds
off on creating, naming the remedy.
(b) A non-mailcow authsource clears the password. Same file:
if ($authsource != 'mailcow'){ $password = ''; $password2 = ''; $password_hashed = '';}The loop still sends a random 32-char password; it costs nothing, it keeps
the create valid if the IdP config is torn down between the gate and the POST,
and the value is discarded by both sides. Mail clients authenticate with app
passwords, minted per consumer by whoever holds the mailcow grant; this
broker mints none.
(c) The Keycloak endpoints exist and behave as assumed.
GET /admin/realms/{realm}/group-by-path/{path} is
@GET @Path("group-by-path/{path: .*}") public GroupRepresentation getGroupByPath(...) in RealmAdminResource.java.
GET .../groups/{id}/members takes first/max/briefRepresentation in
GroupResource.java, and returns direct members only, so a parent group
path will not pick up subgroups (documented in the package README rather than
worked around). GET .../users takes enabled and briefRepresentation among
its query params (UsersResource.java), and brief representation is
id/username/createdTimestamp/firstName/lastName/email/enabled/emailVerified/ federationLink (ModelToRepresentation.java), flat except for the
temp-admin attribute map, which is exactly the nesting the brace-depth splitter
exists to survive.
The mailcow routes (get/mailbox/all/{domain}, add/mailbox,
edit/mailbox with {"items":[…],"attr":{…}}, delete/mailbox with a bare
array) are all already exercised by shipped jobs in apps/email/mailcow/, so
those are in-repo precedent rather than new claims.
3. THE OWNER RULING THAT IS OWED
Section titled “3. THE OWNER RULING THAT IS OWED”What happens to a mailbox when its Keycloak user disappears. The package
ships deletionPolicy: retain and does not need a ruling to be useful; the
creation half is the zero-touch half, and a retain loop is a working loop.
The ruling decides whether an estate may ever set anything else.
| value | effect | reversible |
|---|---|---|
retain (shipped default) | a log line; no write of any kind | n/a |
disable | edit/mailbox {"active":"0"}; mail kept, login and delivery stop; re-enabled if the user returns | yes |
delete | delete/mailbox; mailbox and mail gone, app_passwd cascades | no; only a restic restore of the mailcow stack |
The argument for retain as the default, restated so the ruling is made
against it: a Keycloak user “disappearing” is frequently not a departure. An
admin editing a group, a flapping user-federation link, a realm
export/import, a mistyped groupPath, or an addressSource that filters
everyone out all make users vanish from a listing. Under retain each of
those is a log line. Under delete some of them are data loss, and this
estate has no automated backup on either production estate (memory:
no_backup_schedules_on_either_estate), so “restore the mailcow stack” is not
currently a real recovery path.
Four properties hold under every policy, and they are what would make a
delete ruling survivable if it is made:
- Only ledgered addresses are eligible. A mailbox this loop did not create is never touched. Pre-existing mailboxes are skipped at creation and never adopted into the ledger, so they can never become eligible.
- Grace window.
graceCycles(3) consecutive passes of continuous absence. The counter resets the instant the user reappears and survives restarts. - Mass-absence guard. An empty desired set beside a non-empty ledger refuses the destructive half outright and says why.
dryRunreports every action, creates included, and performs none.
The ruling should also settle the cascade question. Deleting a mailbox
cascades app_passwd; disabling it does not. If a per-user broker (Nextcloud
Mail’s bridge, say) holds credentials for a departed user, disable leaves
those credentials live against a mailbox that no longer authenticates its
owner. Whether the reconciler should also invoke
mailcow-revoke-app-passwords for the departed address, or whether that stays
an operator action, is part of the same decision.
An agent must not flip this var. It is not a step in getting the loop working, and the package’s tests execute the dispatch block once per policy value specifically so that “make it pass” can never mean “make it delete”.
4. What is OWED outside this package
Section titled “4. What is OWED outside this package”(a) apps/VERSION_IMAGE_DIVERGENCE needs one line, or
tests/unit/catalog/app_version_tracks_image_test.go reds; it requires
app.yaml’s version to be a prefix of some rendered image’s tag, and "1.0"
is not a prefix of "8.21.0":
mailbox-reconciler untracked adapter contract version; the only image is the alpine/curl the reconcile loop runs inExactly the exit external-imap and nine other adapters already take.
(b) A keycloakAdmin socket on ce’s keycloak. There is none today,
measured on this tree: 45 distinct *.socket.yaml stems, none of them that.
apps/security/midpoint says so in its own words
(compose.yaml:715, README.md:59) and ships no plug against it, and its
tests assert the absence (tests/tests.bats:355-358, 395). So this package
binds the socket that exists, oidc, and pays two prices:
- It holds master-realm admin credentials to do three read-only things
(
view-users,query-users,query-groups).adminUser/adminPasswordare ordinary published socket data onoidc.socket.yaml, so any consumer of that socket already holds the whole Keycloak; this package does not widen the exposure, it inherits it. - It mints an OIDC client it never uses.
create-oidc-clientbelongs to the socket and runs for every plug bound to it; there is no way to plug this socket and not mint a client. It is minted in the least dangerous shape the socket can produce:publicClient: true(no stored secret; the socket then also setsdirectAccessGrantsEnabled:false) with one absolute, non-routable callback (http://127.0.0.1:1/…; the redirect guard refuses an undeclared field and a bare*, so some URI must be named).
The socket that would fix both: mint a per-consumer service-account client and grant it a NAMED subset of realm-management roles. That replaces a whole-Keycloak credential with a revocable, realm-scoped, read-only one, and turns the collateral client into the credential. midpoint waits on the same socket for the same reason; that makes two consumers, which is the usual bar for building it.
(c) x-shc-scale: max: 1 is declarative. The singleton constraint is real
(two instances would race the same create, and the ledger is a plain file with
no locking) and the engine that would enforce it is not built (memory:
speculative_features_not_in_python). Until it is, “do not scale this service”
is documentation.
5. THE LIVE-PROBE PLAN: what must be proven before this runs near an estate
Section titled “5. THE LIVE-PROBE PLAN: what must be proven before this runs near an estate”Nothing below has been done. Run it on a throwaway mailcow + Keycloak pair,
never on bitspur or ffx, and never with deletionPolicy set to anything but
retain until P6.
Preconditions for every probe: the mailcow stack has its own oidc plug
attached (probe P0 asserts it), and the reconciler is installed with
--var dryRun=true.
| # | Probe | Passes when |
|---|---|---|
| P0 | Install with both integrations, dryRun=true. Read docker logs for one full pass. | The pass line names the right realm, group, domain, and a desired= count that matches the group’s membership. No precondition warning. |
| P1 | The seal. Inspect the deployed vars.yaml and the manifest var_overrides after install; then shc update and docker inspect the container env. | mailcow.apiKey is a ref+vault:// ref after install, and the container env carries the plaintext after the update. The state volume’s api-key is 0600 and matches. |
| P2 | The authsource gate. Detach the mailcow stack’s oidc plug and let one pass run with dryRun=false. | ZERO mailboxes created; the log names the remedy. Re-attach, and the next pass creates. |
| P3 | ★ THE LOAD-BEARING ONE: authsource lands and SSO works. With dryRun=false, let it create one mailbox for a test realm user. Read the row back two ways: GET /api/v1/get/mailbox/<addr> and SELECT authsource FROM mailbox WHERE username='<addr>' in mysql-mailcow. Then log into the mailcow UI as that user via the SSO button. | The DB says keycloak and the SSO login succeeds. If the API projection omits authsource, record that; the package’s post-create warning is best-effort and this probe is the only thing that can measure it. |
| P4 | Idempotence + scale. Let three consecutive passes run over a realm of >200 users (exercises the 100-per-page paginator on a real server). | Pass 1 creates N, passes 2 and 3 create 0 and write nothing. The paginator terminates; desired= equals the realm count. |
| P5 | retain is inert. Remove a test user from the realm/group. Let 5 passes run at the default policy. | The log reports the absence with a rising counter; the mailbox is untouched (active=1, mail intact, app passwords still authenticate). Zero POSTs in mailcow’s own access log for that address. |
| P6 | disable, then return. Only after the §3 ruling permits it. Set deletionPolicy=disable, graceCycles=2. Remove the user; let 3 passes run. Then put the user back; let 1 pass run. | The mailbox goes inactive after pass 2 (not 1), the "active" read-back matches one of the shapes the loop accepts, mail is still on disk, and the return re-activates it. If the read-back shape does not match, the loop will log “did not read back inactive” forever; that is the safe failure, and this probe is what finds it. |
| P7 | delete. Only if the ruling permits it, and only on a mailbox with a known message in it. | The mailbox is gone, its app_passwd rows cascaded, and the ledger row is dropped. Confirm from the mailcow DB, not from the API alone. |
| P8 | The mass-absence guard. Point groupPath at a group that does not exist, then at an empty one. | Missing group → the pass aborts naming the path, nothing changes. Empty group with a non-empty ledger → the GUARD lines fire and the destructive half is skipped under every policy. |
| P9 | Credential failure modes. Rotate the mailcow key on the provider without re-sealing; separately, wrong Keycloak admin password. | Both surface as failing passes with a named cause, the heartbeat stops, and the container goes unhealthy, never a silent no-op and never a crash loop. |
| P10 | Restart + volume loss. Restart the container mid-window; then reinstall with the volume removed. | Restart: counters and ledger survive. Volume loss: the ledger is empty, so the disappearance half stops working and the create half re-converges; it must never delete anything after a volume loss. |
Stop conditions. Any probe that writes to a mailbox the loop did not create
is a hard stop. Any probe where retain produces a write is a hard stop.
6. Two bugs the offline harness already caught
Section titled “6. Two bugs the offline harness already caught”Recorded because they are the shape of what the live probes are for, and because both were silent: a wrong answer, never an error.
- A create was un-ledgered in the same pass.
actualis snapshotted before the creates, and the self-heal drops any ledgered address missing fromactual; every row a pass wrote was therefore deleted again immediately. The ledger stayed permanently empty, so the disappearance half could never act under any policy. Fixed by folding verified creates back intoactualbefore the self-heal; pinned by a test that asserts the ordering. - The JSON extractors assumed space-free JSON. Both servers emit compact JSON today, and every other job in this catalogue assumes it, but the failure mode is not an error, it is an empty desired set, i.e. this broker quietly provisioning nobody while reporting a clean pass. Now whitespace-tolerant, and a pass that sees users but derives no addresses warns and names both likely causes.
Both were found by extracting the rendered command: and running real
reconcile() passes against a stubbed curl. Neither static review nor
shc validate saw either one.