Squashing the migration catalogue, and why it needs no redeploy
★ SUPERSEDED, 2026-08-19, by owner ruling, see
2026-08-19-migrations-reset-doctrine.md. The bridge this document proposed (the manifest, the record-without-executing baseline, the ledger compaction, the flatten guard) was built, shipped in ce, and then removed whole. The owner’s reasoning: a reset is a reset; the entire point of resetting the migration history is that it is NOT safe for an existing deployment; the remedy for an existing deployment is rebuild + external import, never a bridge. This page stays as a dated record of the mechanism as it existed on 2026-08-18. Nothing below is current behaviour:squashes.jsonno longer exists,make migrate/flattenis gone, and the error codes it cites for the bridge (X500973MIGSQHamong them) are retired.
Status: decision document, awaiting ruling. Nothing in this document changes behaviour, and
no migration file was touched to produce it.
Scope: ce/migrations/ (34 Atlas-managed SQL files), the runner in ce/core/db/, and the
question of when the catalogue may be frozen.
Evidence: ce e926c249, shc b40c0714, both estates’ live ledgers read at
level=strong, plus eight executable probes and one full migrate/flatten dry run whose
results are reproduced below.
1. The straight answer
Section titled “1. The straight answer”The backup / redeploy / restore exercise is not required. It was never required.
The ask was:
“get all of our SQL migrations and little Golang migration files all squashed down to a single auto migration… delete the files and run make migrate. However… I’m gonna actually have to back up all of my deployments in [both estates] and completely redeploy and then restore those apps.”
The premise of the second sentence is false. This repository already implements the same baseline mechanism Rails, Django, Flyway and Liquibase use to make a squash survivable: a squashed file is recorded as applied without being executed on any database that already carries its predecessors.
The mechanism is migrations/squashes.json (a map of squashed file → the historical files it
replaces) consumed by db.ApplyMigrations at ce/core/db/migrate.go:180-207:
“Squash baseline: a regenerated catalogue replaces history it squashed. When the ledger shows EVERY predecessor applied, the schema is already at (or past) this file’s content; record it as applied without executing. Without this, the squashed initial schema re-ran against live databases and every node failed to boot with ‘table already exists’ (observed on the proxmox cluster after a squash).”
The file exists, is embedded in the binary (migrations/embed.go, //go:embed … squashes.json),
is loaded on the production path (subsystems/impl/migrations.go → db.MigrateFS →
db.LoadSquashes), and today contains {}.
So the whole exercise reduces to:
| the owner’s plan | what is actually required |
|---|---|
| back up every deployment on both estates | nothing |
| tear both estates down | nothing |
| redeploy from scratch | nothing |
| restore every app from archive | nothing |
| none | one repo commit: delete 34 files, add 1, populate squashes.json, re-hash atlas.sum |
| none | roll the resulting release; each estate records one extra ledger row and executes no SQL |
One thing in that commit is load-bearing and one thing only: squashes.json must be populated
in the same commit that deletes the files. A squash that ships with squashes.json still {}
re-runs the baseline against live databases and bricks every node with “table already exists”.
That is the entire risk surface, it is a single JSON file, and section 4 gives the exact contents.
2. What was measured
Section titled “2. What was measured”2.1 The runner does what it claims: eight probes
Section titled “2.1 The runner does what it claims: eight probes”Probes were compiled against the real core/db package and deleted afterwards; the clone is
byte-clean at e926c249. Five ran with the dev escape hatch on, three with production
verification (atlas.sum + per-row ledger checksums) fully enabled.
| # | scenario | result |
|---|---|---|
| 1 | full history applied, then the squashed catalogue | baselined; one row inserted, no SQL executed, the pre-existing column added by an ALTER survives untouched |
| 2 | squashed catalogue against a virgin database | executed in full; fresh installs get the whole schema |
| 3 | a node holding only 2 of 3 predecessors | refused, loudly: X500973MIGSQH … conflicts with an existing schema |
| 4 | old files deleted from the catalogue, names still in the ledger | no drift error; reboot idempotent; a later migration stacks on top cleanly |
| 5 | reusing an already-applied filename as the squash name | accepted, because the escape hatch disables ledger checksum enforcement |
| 1v | probe 1 again, production verification on | baselined; reboot idempotent; the baselined row’s stored checksum matches the shipped file forever after |
| 5v | probe 5 again, production verification on | refused: X500985MIGLDG … applied migration drifted, ledger checksum … but on-disk content is … |
| 6v | an estate that skipped the squash release and takes squash + next in one step | baselined, then the next migration ran |
Probe 3 is the pre-flight requirement (section 4.2). Probe 5/5v is the naming trap (section 4.3). Probe 6v matters because it means an estate does not have to stop on the squash release.
2.2 The flattened baseline is semantically identical to replaying history
Section titled “2.2 The flattened baseline is semantically identical to replaying history”make migrate/flatten was run in a throwaway copy of the repo. It produced one file,
20260818023245_squash.sql: 1,040 lines / 46 KB / 44 CREATE TABLE / 167 indexes, replacing
34 files / 2,663 lines / 146 KB.
Both catalogues were then applied to separate in-memory databases and compared, not by DDL text
but by semantics: every column’s name, type, nullability, default and primary-key flag, every
foreign key with its ON UPDATE/ON DELETE, and every index with its uniqueness and column list:
tables identical: 44objects: replayed=211 flattened=211ONLY IN REPLAYED HISTORY: 0ONLY IN FLATTENED BASELINE: 0SEMANTIC MISMATCHES: 0PHYSICAL COLUMN-ORDER DIFFERENCES (cosmetic): 7The seven are container, deployment, node, service, store, volume,
volume_attachment, tables whose history includes an ALTER … ADD COLUMN that appended where
the flattened generator interleaves. Nothing in the tree depends on ordinal position:
SELECT * appears zero times in non-test Go source, as does any column-list-less INSERT.
Ent always names its columns. The divergence is real but inert, and it is permanent; the two
estates keep the replayed order forever while every public install gets the flattened order.
2.3 No seed data is lost
Section titled “2.3 No seed data is lost”A generated baseline is DDL only, so anything a fresh install needs that is not DDL would
vanish. Twelve of the 34 files carry INSERT/UPDATE/DELETE statements, which looked like a
serious hazard until each was read. Every single INSERT in the catalogue is an
INSERT … SELECT table-rebuild carry. There is not one literal VALUES seed anywhere in the 34
files, and every UPDATE is a backfill of existing rows (WHERE … IS NULL, a value rename, a
de-duplication). A database with no rows needs none of them.
That is what makes the flatten sound, and it is the same invariant
TestFromEmptySchemaEqualsTheMigratedSchema asserts today. That test must survive the squash in
adapted form; it is the guard that keeps the ent schema and the catalogue honest.
2.4 Both estates are exactly where they need to be
Section titled “2.4 Both estates are exactly where they need to be”Both estates carry byte-identical 34-row ledgers, sha256 a3b74dc4…, newest row
20260817170000_addon_rename.sql. There is no divergence to reconcile, and every predecessor the
baseline will name is present on both. The compatibility window for this change is therefore
exactly two databases, both already current, which is the whole argument for doing it before
launch rather than after.
3. The redeploy path is not merely unnecessary; it is unsafe
Section titled “3. The redeploy path is not merely unnecessary; it is unsafe”Stated plainly, because the owner’s plan bet both production estates on it. The restore path he intended to use has three confirmed defects on current main:
| defect | consequence |
|---|---|
multi-node restore recovers into the wrong environment, ignores --node, and leaves the deployment marked UNUSABLE | a restore that “succeeds” puts data somewhere it was not asked to go and marks the result unusable |
--paths is inert: accepted, parsed, never consumed | ask for a subset, get everything |
| archive checksums are written but never compared on read | a corrupted archive restores silently and looks fine |
Independently, docs/decisions/2026-08-16-durability-not-sagas.md already ranks
“recover leaves a live, healthy-looking, EMPTY stack” as the third-worst reachable state in the
system, and “restore stranded mid-wipe” as the fourth.
Because the baseline path never restores anything, these stop being blockers for this decision. They remain blockers for any future exercise that does rely on restore, and they are the reason the answer to “should I rehearse a full estate rebuild to de-risk the squash” is no; the rehearsal is riskier than the thing it would de-risk.
4. The procedure, if the ruling is to proceed
Section titled “4. The procedure, if the ruling is to proceed”4.1 The change
Section titled “4.1 The change”cd ce && make migrate/flatten I_UNDERSTAND_FLATTEN_BREAKS_LIVE_ESTATES=1 NAME=squashdeletesmigrations/*.sqlandatlas.sum, regenerates one baseline froment/schema, and re-hashes. Verified working; output measured in §2.2.- Populate
migrations/squashes.jsonin the same commit. Contents below, with the generated filename substituted for the placeholder. All 34 predecessors, strictly; a shorter list would let a database that is genuinely behind baseline onto a schema it does not have. - Delete or adapt the 21 test files in
ce/migrations/(5,545 lines, 43 funcs). Five hard-code SQL filenames and break the moment the files go, includingapplication_backfill_test.go, which carriesTestFromEmptySchemaEqualsTheMigratedSchema, the invariant that makes squashing sound at all. Adapt that one against the baseline; do not drop it. - Do not touch
embed.go,core/db/migrate.go, orsubsystems/impl/migrations.go. No code change is required anywhere.
{ "<GENERATED_BASELINE>.sql": [ "20260727022830_auto.sql", "20260728080502_tenant_tombstone.sql", "20260730032524_node_cpu_cores.sql", "20260731192534_fabric_posture_pins.sql", "20260802033902_node_voter_slot.sql", "20260802201225_service_resource_columns.sql", "20260802201638_deployment_pinned_node.sql", "20260803002303_container_exit_cause.sql", "20260803090000_vault_envelope_only.sql", "20260803150452_deployment_app_digest.sql", "20260803152000_service_quota_columns.sql", "20260803170107_connection_tombstone.sql", "20260804022217_auto.sql", "20260804050215_deployment_node_set.sql", "20260804170000_oidc_socket_rename.sql", "20260804183840_vdeployment.sql", "20260804183842_app_browse_columns.sql", "20260804185355_preset_rows.sql", "20260804200106_app_images_recorded.sql", "20260804221823_app_requirement_declaration.sql", "20260804222619_usage_vdeployment_count.sql", "20260805004250_application_table.sql", "20260805031927_drop_deployment_instance_name.sql", "20260805054457_drop_deployment_parent_index.sql", "20260807042540_task_affinity_node.sql", "20260807064858_node_mesh_ip_unique.sql", "20260807170745_auto.sql", "20260807223000_backup_schedule_tier_key.sql", "20260812023430_store_registry_and_seed.sql", "20260814200549_node_architecture.sql", "20260815060000_datetime_text_utc_format.sql", "20260815070000_deployment_lease.sql", "20260815182958_volume_attachment_state_reason.sql", "20260817170000_addon_rename.sql" ]}4.2 Do not start unless
Section titled “4.2 Do not start unless”- Both estates read exactly the 34 names above from
_shc_migrationsvia the live rqlite API atlevel=strong. The on-diskdb.sqliteis a lagging raft snapshot and must not be used for this check. A missing predecessor means the baseline executes and the node bricks (probe 3). -
squashes.jsonis non-empty in the diff. If it is{}, stop. - The generated baseline filename is new; it must not appear in either ledger (§4.3).
-
atlas.sumis regenerated and lists exactly the one file. -
TestFromEmptySchemaEqualsTheMigratedSchemapasses in adapted form. - The two guard tests in §6 exist and pass.
-
feat/payload-interninghas been ruled on (§5.3). - CI is green. Do not merge red.
4.3 Traps, each one measured
Section titled “4.3 Traps, each one measured”The baseline filename must be new. Reusing an existing name (the obvious instinct, “make the
first migration the baseline”), is fatal: the runner finds the name already in the ledger, compares
the stored content checksum against the new file, and refuses to boot with X500985MIGLDG
(probe 5v). atlas migrate diff stamps a fresh UTC timestamp, so following §4.1 avoids this
automatically. Hand-editing the filename afterwards reintroduces it.
Roll nodes one at a time, leader first. There is no leader gate, no advisory lock and no
INSERT OR IGNORE around the ledger write (core/db/migrate.go:201,243). Every node runs the
runner at boot against the shared raft database. Two nodes booting together can race the single
baseline INSERT and one will lose on the primary key. Restart=always / RestartSec=5 in
shc.service means the loser recovers in about five seconds (this is a transient, not a brick)
but serialising removes the noise entirely. Wait for the new ledger row before moving to the next
node.
The ledger is per-cluster, not per-node. Migrations run against the raft-replicated handle after the engine swap, so each estate has one ledger, not one per node. The 7-node, 2-estate framing collapses to two databases.
Verify by reading the ledger, not by exit status. Restore is not the only thing in this estate that exits 0 without having done the job. The proof that the roll worked is:
SELECT name FROM _shc_migrations ORDER BY name;35 rows on each estate, the 34 originals untouched plus the baseline, and every deployment still
running. If the row count is 34, the release did not take. If any node logs X500973MIGSQH, stop
the roll: that node’s database was behind and §4.2 was not honoured.
4.4 Rollback
Section titled “4.4 Rollback”Rolling back is a package downgrade and nothing else. The previous release ships all 34 files;
the ledger still has all 34 rows; the orphan baseline row sits lexically after them and outside
the drift check’s window, so assertNoDrift does not fire (probe 4). If a belt-and-braces
cleanup is wanted, delete the one row through the rqlite API. No schema is ever mutated by this
change on an existing estate, so there is nothing to reverse.
4.5 The manual escape hatch
Section titled “4.5 The manual escape hatch”If squashes.json is ever wrong in a shipped build, the baseline can be recorded by hand, per
estate, through the live rqlite API:
INSERT INTO _shc_migrations (name, checksum)VALUES ('<baseline>.sql', 'sha256:<sha256 hex of the file body>');The checksum is db.ContentChecksum, plain sha256 of the file contents with a sha256: prefix.
This is the “one-row insert per estate” fallback. It should never be needed; it exists so that a
mistake is a two-minute fix rather than an outage.
5. Is the schema ready to be frozen?
Section titled “5. Is the schema ready to be frozen?”The honest answer: converging fast, not yet frozen, and three known migrations are still owed.
5.1 The rate is down 62% from peak
Section titled “5.1 The rate is down 62% from peak”Measured from the UTC timestamps in the filenames:
| window | migrations |
|---|---|
| Jul 27 – Aug 3 | 12 |
| Aug 4 – Aug 10 | 16 (peak) |
| Aug 11 – Aug 18 | 6 |
Supporting evidence, all pointing the same way:
- Across 217 commits in the last three days, exactly one touched
ent/schema/. - 1 of 77 ce merge requests merged in the last 21 days added a migration.
- Zero open merge requests in either project touch
ce/migrations/orce/ent/schema/. - ce!88 (backup console editor, fork-on-write lineage, the newest schema-adjacent workstream) changes 13 files and lands no schema at all.
- Longest quiet streak observed to date: 4 days (Aug 8–11).
5.2 The convergence is architectural, not luck
Section titled “5.2 The convergence is architectural, not luck”This is the part that should raise confidence rather than merely describe it. New persisted state
is being deliberately routed into the already-migrated generic kv and config tables rather than
into new ones: auth_session_revocation, launcher.layout, portal.registration,
license.estate all live there, a pattern stated flatly in shc/modules/portal/registration.go.
Features that would once have cost a migration now cost a key. That is why several of the
workstreams that sounded like guaranteed schema (credential revocation, launcher_placed on the
console wire, install reporting) produce none.
5.3 Predicted migrations still to come
Section titled “5.3 Predicted migrations still to come”| # | migration | confidence | note |
|---|---|---|---|
| 1 | installed_by-style column | ✅ LANDED 2026-08-18 | ce!114, migrations/20260818140325_installed_by.sql: two plain ADD COLUMNs (deployment, vdeployment), no backfill, no index, no rebuild vocabulary. Shape recorded in 2026-08-16-marketplace-identity-and-install.md §9.1 |
| 2 | repo.origin → repo.adapter rename | ✗ NEVER OWED; struck 2026-08-18 | the column has been spelled adapter since the FIRST migration. This row was a prior audit misreading a CLI verb rename as a column rename. Do not re-add it |
| 3 | ✗ NO MIGRATION; resolved 2026-08-18 | the lineage rides the existing manifest Text column. This row’s own note anticipated it: “ce!88 lands none today, which is evidence the lineage may fit in existing columns” | |
| 4 | feat/payload-interning | ✅ MERGED as ce!111, 2026-08-18 | landed on main as migrations/20260818071711_payload_interning.sql. The choice this row said a squash would force was made BEFORE the squash: rebase, not abandon |
| 5 | marketplace community layer (reviews, ratings, threaded comments, votes, install counts) | none; wrong system | see §5.4 |
| 6 | credential revocation · launcher_placed · install reporting · backup and exec permission gates · consensus guardrails | low | surveyed; each lands in kv/config or is derived, not persisted |
RESOLVED 2026-08-18, merged as
ce!111. It is on feat/payload-interning is the one genuinely unresolved item.main as migrations/20260818071711_payload_interning.sql (a payload_blob
table plus payload_ref columns on events_log and audit), so the decision this paragraph said a
squash would force (rebase onto the new baseline, or abandon) was taken before the squash and
in the cheaper direction. The paragraph is kept because its argument still holds for the next parked
branch: a squash does not create that choice, it merely stops letting anyone defer it.
The original finding, for the record: an unmerged branch parked ten days carrying a 35th migration
20260808080553_auto.sql, with no merge request ever opened, a stated ≥72-hour soak gate that
had not been run (docs/internal/payload-interning-soak-runbook.md), and a changelog recording it
deferred for three consecutive release cycles.
5.4 The community layer is a different migration system; say this to users
Section titled “5.4 The community layer is a different migration system; say this to users”The marketplace community layer (reviews, ratings, threaded comments with voting, report flows)
is Frappe DocTypes. It migrates through bench migrate, inside the Frappe app container,
against that app’s own database. It generates zero files in ce/migrations/ and is completely
untouched by anything in this document.
This matters beyond bookkeeping. The owner’s concern is “I don’t really want users having loads
of migrations they’re gonna have to go through.” A user’s real migration burden is the sum of two
independent systems, and squashing one says nothing about the other. Freezing ce/migrations/
perfectly still leaves the Frappe side migrating on its own cadence. Any public claim about
migration burden has to name both systems or it will be wrong.
5.5 What “delete the Golang migration files” means: it is three different things
Section titled “5.5 What “delete the Golang migration files” means: it is three different things”| category | count | what happens |
|---|---|---|
tests in ce/migrations/ | 21 files, 43 funcs, 5,545 lines | deleted or adapted. Five hard-reference SQL filenames and break the moment the files go, and one of the five, application_backfill_test.go, is where TestFromEmptySchemaEqualsTheMigratedSchema lives. It must be adapted, not merely kept |
runtime backfills outside migrations/ | 7 files, 1,688 lines | mostly survive; they migrate on-disk state, not schema. But ce/modules/app/vdeployment_dir_migration.go:81-82 states outright: “What WOULD license removal: a flatten, or a daemon-enforced release floor at >= v0.0.22.” The squash unlocks deletions that are currently blocked |
the runner itself (core/db/migrate.go, verify.go, pool.go, subsystems/impl/migrations.go) | 4 files | stays. Untouched. |
5.6 The trigger condition
Section titled “5.6 The trigger condition”Not a date, a state. The window opens when all of the following hold:
-
Ten consecutive days with no new file in
ce/migrations/. The longest quiet streak so far is four days, so ten is roughly 2.5× the observed noise floor and is a real signal rather than a lull. -
The three owed migrations have landed:
✅ ce!114, theinstalled_byrepo.origin→repo.adapterrename, and whatever backup lineage finally needs (or a written finding that it needs nothing).Two of the three are now struck rather than done, and the difference matters. The
repo.origin→repo.adapterrename was NEVER OWED: the column has been spelledadaptersince the first migration, and this row was a prior audit misreading a CLI verb rename as a column rename. Backup lineage rides the existing manifestTextcolumn, the finding this condition allowed for. So after ce!114 this gate is CLEAR, and the long pole is condition 3 alone, which ce!111 also closed by mergingfeat/payload-interning. -
feat/payload-interningis ruled on: merged, or abandoned and its branch deleted. -
Both guard tests from §6 exist and pass.
-
Both estates read the full predecessor list at
level=strongon the day of the change.
Condition 1 is the clock; 2–5 are gates. On the current trajectory conditions 2 and 3 are the long poles, not the cadence.
One correction to the framing of the original ask, and it changes the stakes entirely. The owner said “I don’t wanna have to continuously go through it again and again.” With the baseline mechanism, a second squash costs exactly what the first one costs: a commit and a ledger row. The dread is proportionate to the redeploy plan, not to the change. There is no reason to hold the squash hostage to certainty that no migration will ever follow, only to the three that are already known to be coming, because those are cheap to wait for.
6. Owed work, small, and it should land before the squash
Section titled “6. Owed work, small, and it should land before the squash”Two guard tests, both absent today:
- The baseline branch has zero test coverage. Every test in the tree passes
db.LoadSquashes(migrations.FS), which is{}, so the branch that makes this entire decision safe has never been executed by CI. The probes in §2.1 are the shape such a test should take. This is the single highest-value item in this document. squashes.jsonis unvalidated. A typo in a predecessor name degrades silently to “no squash”; the baseline then executes and bricks the estate. Nothing checks it. A test asserting that every name listed matches the set of files deleted in the same commit would convert the only real failure mode into a red pipeline.
One CI gap worth naming: nothing asserts that ent/schema and the migration head agree.
grep 'migrate diff' .gitlab-ci.yml returns nothing, so a schema edit merged without its migration
is caught only at boot, on an estate.
And a ruling is requested on the flatten guard. make migrate/flatten is disarmed behind
I_UNDERSTAND_FLATTEN_BREAKS_LIVE_ESTATES=1 by the owner’s own 2026-07-30 append-only ruling, a
ruling made before the baseline mechanism existed, and whose premise (“this destroys applied
migration history”) is no longer true when squashes.json is populated. The recommendation is not
to remove the guard but to re-arm it against the invariant that actually matters: refuse unless
migrations/squashes.json is non-empty and names every file the flatten is about to delete. That
turns a variable asserting an intention into a guard asserting a fact, which is the pattern the
target already argues for in its own comment.
7. Recommendation
Section titled “7. Recommendation”Proceed with the squash as a repo change. Do not back up, do not redeploy, do not restore.
Land the two guard tests first, wait for the three owed migrations and a ruling on
feat/payload-interning, then take the ten-day quiet window whenever it next appears. The change
is one commit, one release, and one extra row in each of two ledgers.
The exercise the owner was bracing for (the one he described as “a huge change”) is a
git rm, a JSON file, and a dpkg -i.