Skip to content
SHC Docs

Migrations

This workflow runs in the ce repo, not here. The split moved ent/, migrations/ and atlas.hcl there, and the make migrate target moved with them; every command below is run from a ce checkout. This repo consumes ce as a pinned Go module and the daemon applies the embedded migration at startup, so there is nothing schema-related to run on the shc side. File paths below are ce-repo-relative.

SHC uses Atlas (ariga/atlas) with Ent (entgo.io/ent) as the schema source. Tables are declared as Go structs under ent/schema/. make migrate NAME=<slug> diffs the catalogue against those declarations and writes the next migration. The SHC daemon applies the catalogue at startup through the runner at core/db/migrate.go.

make migrate needs the Atlas CLI (make prepare). Operators don’t need Atlas on production hosts.

The store starts from one first file (SHC-2197), Atlas’s plan from an empty database to the tables ent declares. A schema change appends a numbered migration on top of it. make migrate is the only migration target in ce.

ent/
generate.go # go generate ./ent entrypoint
entc.go # codegen driver
schema/
<table>.go # one file per table
<table>.go, <table>_create.go, … # generated typed client (don't edit)
core/db/released.sum # atlas.sum lines of every migration a release shipped
migrations/
<UTC-timestamp>_auto.sql # the first file
<UTC-timestamp>_<slug>.sql # appended by make migrate NAME=<slug>
atlas.sum # integrity manifest for all of them
embed.go # //go:embed *.sql atlas.sum

ent/schema/*.go is the source of truth. It drives both the typed CRUD client (used by every module’s repo via core/entclient) and the migrations. Nobody edits a migration or atlas.sum by hand.

  1. Edit ent/schema/<table>.go (add, remove or change fields, edges, indexes, CHECKs).
  2. Run go generate ./ent to regenerate the typed client.
  3. Run make migrate NAME=<slug>. Atlas diffs the catalogue against ent and writes migrations/<UTC-timestamp>_<slug>.sql and a new atlas.sum.
  4. Commit the schema change, the client and the new file together.
Terminal window
$EDITOR ent/schema/deployment.go
go generate ./ent
make migrate NAME=deployment_domain

An appended migration is additive. A live cluster takes a new build one node at a time, so the previous build still runs against the store the new one migrated: a new table, a nullable or defaulted column and a new index are fine. Removing or reshaping something the previous build reads takes two releases. The first stops reading it, the second removes it and names the first in a line -- shc:last-reader-removed-in vX.Y.Z.

A merge conflict in atlas.sum is resolved by taking either side and running atlas migrate hash (the first line of make migrate). Two branches that both appended keep both files.

core/db/released.sum lists the atlas.sum line of every migration a release has shipped. The release commit writes it in ce, from the catalogue the tag ships:

Terminal window
grep -v '^h1:' migrations/atlas.sum >core/db/released.sum

TestEveryReleasedMigrationShipsUnchanged in tests/_ce/migrations fails when a recorded line is missing from the catalogue or hashes differently, so a shipped migration is never edited, renamed or removed. A release tag’s pipeline refuses a pinned ce whose released.sum is not exactly the file lines of its atlas.sum (release/freeze-check, scripts/ci-freeze-check.sh), so a tag cut without the record fails its own pipeline.

A reset replaces the catalogue with one new first file. It ships in a release whose clusters are reinstalled, never upgraded. In ce, on that day’s dev: rm migrations/*.sql && make migrate, then move ResetLine in core/db/reset_line.go to the reset release, and rewrite core/db/released.sum from the new atlas.sum in the same commit. That commit is the one sanctioned change to a released line.

atlas migrate diff against ent://ent/schema is not empty on today’s catalogue: ent names an index past 64 characters <first 31 characters>_<md5 of the name> at runtime, which is what the first file holds, while Atlas’s ent loader keeps the long name. Today that is integration_tenant_name_consumer_deployment_id_consumer_plug_name and replicalabel_tenant_name_environment_stack_name_service_name_ordinal_key. Anything else in that diff is real drift.

The drift tests live in the shc repo’s test store, tests/_ce/migrations/ (run overlaid into ce), and tests/unit/migrations/:

  • TestEveryReleasedMigrationShipsUnchanged: every line of core/db/released.sum is in the catalogue, byte for byte.
  • TestEveryMigrationAfterTheFirstFileIsAdditive: the additive rule above.
  • TestFromEmptyMigrationsMatchTheEntDesiredState and TestEntSchemaAndMigrationHeadAgree: the store the catalogue builds from empty is the store ent declares, table by table and column by column.
  • TestEntIndexNamesAreTheNamesMakeMigrateWrites: no index name ent declares is over 64 characters. ent’s Create hashes a longer name and make migrate writes it in full, so give such an index a StorageKey of at most 64 characters.

What a new first file does to an existing store

Section titled “What a new first file does to an existing store”

A store remembers the files it was made from in its _shc_migrations ledger. A build refuses, at start and before any SQL runs, a store whose ledger names a migration it does not ship: X409990MIGRST when the store was made from a history this release no longer ships (“reinstall from this release”), X409991MIGNEW when a newer build migrated it. A store made from the first file and earlier appended migrations is upgraded in place. A store made before a reset is not: the cluster is reinstalled from the reset release and its apps’ data brought back from backups.

A reset moves the reset line, ce core/db/reset_line.go ResetLine, to the reset release. A node records the line it was formed on, and shcd refuses one formed before its own with X409992PRELIN, exit 65, before anything starts (see reset lines).

Every ent/schema/<table>.go:

  • Type name = table CamelCase (mesh_token → MeshToken, store_node → StoreNode).
  • Annotations() pins the SQL table name explicitly via entsql.Annotation{Table: "<name>"}; Ent pluralizes by default.
  • VARCHAR(N) UUID PKs: redeclare via field.String("id").MaxLen(N).Immutable().
  • VARCHAR(N) non-UUID PKs: use field.String("id").StorageKey("<col>").Immutable().
  • Field names match column names verbatim (snake_case); Ent derives Go field names from those.
  • A closed set of values carries a CHECK in the table’s annotation.
  • Edges: parent declares edges.To("children", Child.Type) with Annotations(entsql.OnDelete(...)); child declares edges.From("parent", Parent.Type).Ref("children").Field("col").Unique().

The canonical exemplar lives in ent/schema/kv.go.

rqlite replicates plain SQLite statements over raft, so the one file is plain SQLite: CREATE TABLE and CREATE INDEX from an empty store. It never sends a PRAGMA and never drops a table, which tests/_ce/migrations/catalogue_shape_test.go holds.