The occupancy gate stands on the wrong column, and a join table is the fix
Superseded by New-line runtime. The text below is the record as of its date.
Status: design settled, ready to build: BUILT AND SHIPPING IN
v0.0.19. ce 397823f implements this document as written: the
deployment_node set, the new OccupancyLister port the gate reads, the
four-times floor, the positive-evidence prune rule, the single-valued anchor,
the fail-closed refusal with no fallback to DeploymentLister, and the
additive migration with the node_id IN (SELECT id FROM node) filter §6
called load-bearing. recorded_node_id stayed, narrowed to routing, exactly
as §5 ruled. The v0.0.19 pin bump carries it, so it changes estate behaviour
from the next roll onward.
shc’s half of the landing is the test surface, in the same release: the route-level guards moved onto the occupancy port, the fail-closed matrix stopped being discharged by one unwired port, and two assertions were added that this repo did not have: that the gate refuses a node whose containers are observed here while the anchor listing answers zero (the false negative §1 names as the data-loss direction), and that it does NOT refuse on the anchor listing alone.
Verdict on the owner’s question, unchanged: he is right about the gate and
right about the join table; he is wrong about recorded_node_id, which has a
real surviving consumer and cannot go.
Baselines. ce origin/main 852c768, shc origin/main b4aed8e42.
The shipped design under review is ce d65b64b, in v0.0.18. Every figure
below was measured against those trees. No estate, cloud, DNS or pipeline
was touched.
1. The verified defect: the gate is unsound in BOTH directions, and the
Section titled “1. The verified defect: the gate is unsound in BOTH directions, and the”dangerous one is the false negative
Section titled “dangerous one is the false negative”v0.0.18 split deployment.node_id into recorded_node_id (the rename;
NOT NULL, RESTRICT FK to node) and pinned_node_id (new, nullable,
FK-less, placement intent). The node-removal occupancy gate was deliberately
left standing on recorded_node_id, “untouched”. That is the defect.
The chain, read end to end:
- What gets written.
modules/stack/service.gorecordStackRow:Node: firstNonEmpty(req.NodeID, target.NodeID). Under swarm with no--node,req.NodeIDis empty, so the value stored istarget.NodeID, the node that handled the install request. - Where the tasks actually go. An unpinned swarm install stamps no
placement constraint at all.
install_runner.go:1169-1200handsStampPlacementNode: opts.PlacementNode(the operator’s--nodeas a node name), andswarm_placement_replay.gois explicit that the recorded node is “deliberately NOT an input here” and that “an unpinned swarm deployment stays unpinned”. So swarm schedules the tasks on any eligible node. - What the gate counts.
modules/app/deployment_repo.go:112-123:SELECT … FROM deployment WHERE deployment.recorded_node_id = ?.cluster.Service.DeploymentsOnNode→EnsureNodeUnoccupied→X400017NODDEP.
Therefore, for a swarm deployment installed from node A with tasks scheduled on B and C:
| removing | rows returned | gate | truth |
|---|---|---|---|
| A | 1 | REFUSES | no tasks here; only the deployment directory |
| B | 0 | PERMITS | live tasks and their local volumes are here |
The false positive on A is an annoyance. The false negative on B is
the data-loss direction, and the removal is not a paper one:
modules/cluster/swarm_removal.go demotes-then-runs docker node rm --force
(swarm_managers.go:491), so shc cluster rm B tears a live worker out of
the swarm while SHC reports it had nothing on it. Any local-volume service
whose task was running there is orphaned.
The worst instance is the automatic path. All three “remove another node”
routes discharge this one gate (shc modules/cluster/admin_router.go:88 for
the cluster repair sweep, :239 for cluster rm,
lifecycle_router.go:347 for forget); the routes are EE, so ce’s own
occupancy.go has no ce-side caller and grepping ce alone makes the gate look
dead. cluster rm --force skips the gate by design and keeps doing so; the
sweep and forget cannot. The repair sweep runs without an
operator in the loop, and its own comment says so: “this is the AUTOMATIC
removal path, so a 24-hour partition evicted a node carrying live stacks with
no operator in the loop at all.” Under swarm it still does, for every node
that is not the install node.
The code already admits the true positive is unsound too.
modules/cluster/occupancy.go OccupancyBasis renders, for swarm rows:
“under swarm, where placement is swarm’s; this node is only what SHC recorded at install time, and SHC did not establish where the tasks or data are now”
A gate whose own refusal text says it established nothing about placement is
not a placement gate. v0.0.18 made the rendering honest and left the
decision wrong.
2. The rebuild argument does not reach a join table: MEASURED
Section titled “2. The rebuild argument does not reach a join table: MEASURED”d65b64b’s hand-written migration is correct and its note must stay. Atlas
generates a table rebuild for the deployment column change, and against
rqlited v10.2.7 started with -fk (as core/rqlited does) with statements
submitted as one transaction (as core/db.applyOne does) that rebuild
measured deployment 3→1, service 1→0, host_mapping 1→0: DROP TABLE
performs an implicit DELETE when foreign keys are enforced, so every
ON DELETE CASCADE child is destroyed.
That measurement justifies the handling of the COLUMN change. It says
nothing about a new table, and extending it to one was the error in the
shipped design’s reasoning. Measured, not reasoned about; the join-table
ent schema of §3 was scaffolded into a fresh ce worktree and make migrate
run against it. Atlas emitted exactly this, and nothing else:
-- Create "deployment_node" tableCREATE TABLE `deployment_node` ( `id` integer NOT NULL PRIMARY KEY AUTOINCREMENT, `anchored` bool NOT NULL DEFAULT false, `first_seen_at` datetime NOT NULL, `last_observed_at` datetime NULL, `deployment_id` integer NOT NULL, `node_id` text NOT NULL, CONSTRAINT `deployment_node_node_occupied_by` FOREIGN KEY (`node_id`) REFERENCES `node` (`id`) ON UPDATE NO ACTION ON DELETE RESTRICT, CONSTRAINT `deployment_node_deployment_occupancy` FOREIGN KEY (`deployment_id`) REFERENCES `deployment` (`id`) ON UPDATE NO ACTION ON DELETE CASCADE);-- Create index "deploymentnode_deployment_id_node_id" to table: "deployment_node"CREATE UNIQUE INDEX `deploymentnode_deployment_id_node_id` ON `deployment_node` (`deployment_id`, `node_id`);-- Create index "deploymentnode_node_id" to table: "deployment_node"CREATE INDEX `deploymentnode_node_id` ON `deployment_node` (`node_id`);No PRAGMA. No DROP TABLE. The deployment and node tables are not
emitted at all; adding the inverse edges on their side moves no bytes,
because the FK columns live on the child. migrations/catalogue_shape_test.go
and core/db/rebuild_guard_test.go both pass on the file as generated.
There is precedent in the schema for exactly this shape:
ent/schema/store_node.go is already a (store_id, node_id)
join table with two FK edges, and 20260803170107_connection_tombstone.sql
is an Atlas-generated additive CREATE TABLE in the live catalogue.
3. Target schema
Section titled “3. Target schema”Three facts, three homes. The shipped design has two of them right.
| fact | question | home | shape |
|---|---|---|---|
| INTENT | where did the operator ask for it? | deployment.pinned_node_id | scalar, nullable, no FK; unchanged, already correct |
| OCCUPANCY | which nodes hold its bytes or run its tasks? | deployment_node (new) | a SET, always non-empty for a live deployment |
| ROUTING | which single daemon executes its verbs? | deployment.recorded_node_id | scalar, NOT NULL, RESTRICT FK; kept, narrowed (§5) |
deployment_node
Section titled “deployment_node”field.Int("deployment_id") // → deployment, CASCADEfield.String("node_id").MaxLen(255) // → node, RESTRICTfield.Bool("anchored").Default(false) // SHC wrote bytes to this node's diskfield.Time("first_seen_at")field.Time("last_observed_at").Optional().Nillable() // NULL = never observed// index.Fields("deployment_id", "node_id").Unique(); index.Fields("node_id")// edge.From("deployment", …).Ref("occupancy").Field("deployment_id").Unique().Required()// edge.From("node", …).Ref("occupied_by").Field("node_id").Unique().Required()Two independent facts per row, not one enum. anchored means SHC put
bytes on this node’s filesystem: the rendered compose, the state manifest,
the deployment directory. last_observed_at means SHC has seen containers of
this deployment here. A node can be either, both, or (transiently) neither.
Collapsing them into one origin column forces a lossy choice for the node
that is both, which is the ordinary compose case.
This is the thing neither shipped column can be. It is always
non-empty for a live deployment, because the anchor is written by the same
install path that writes recorded_node_id, so the gate keeps its floor.
And it is correct under swarm, because observation adds and removes rows.
recorded_node_id has the floor without the correctness; a derived-only set
has the correctness without the floor.
Derived-only is not survivable, and the reason is in the tree. The observed set legitimately reads EMPTY for a running or restorable deployment in at least three shapes:
- A stopped deployment.
shc stopremoves the containers; the leader reaper (container_reconciler.go,updated_at < cutoff, 120s) deletes the rows.ObservedNodesthen returnsnilwhile the compose deployment directory and every local volume still sit on that node’s disk. - An unreachable node. Its recorder stops heartbeating
(
TouchContainers), the leader reaps its rows after 120s, and the derived set empties, during the exact outage that makes removal dangerous. This is thecluster repairsweep’s whole subject class. - The window before first observation. A fresh install has no container rows for up to one recorder pass.
A gate that derives to zero in those three cases is a gate that permits removing a node that holds live state. The anchor row is what makes the answer never zero.
pinned_node_id stays exactly as it is. Intent is a different question
from occupancy, its FK-lessness is deliberate (a pin naming a removed node is
information, not corruption), and app.PlacementIntent already owns the one
rule that fills it. Nothing here touches it.
4. What maintains the set
Section titled “4. What maintains the set”Four writers, and only four.
- Install / re-render: the
recordStackRow→UpsertStackpath. UPSERT the row for the resolved node withanchored = 1. Idempotent. Written from the same call that writesrecorded_node_id, so the two cannot disagree about which node materialized the deployment. - Observation: a leader-only occupancy reconcile, folded into the
existing
ContainerReconciler(60s, already leader-only, already authorized to delete container rows). Per deployment: UPSERT a row for every node the container-derived set names, stampinglast_observed_at; then DELETE non-anchored rows the derivation does not name. - Move / drain:
app.DeploymentRepo.Migrate. Setanchored = 0on the old node’s row (leavinglast_observed_atalone), UPSERT the target’s row withanchored = 1. The pin rule already there is unchanged: a move follows an existing pin and never creates one. - Uninstall: nothing to write.
ON DELETE CASCADEfromdeploymentclears the set. MEASURED (§7): deleting a deployment left zero occupancy rows.
★ THE PRUNE RULE: prune on POSITIVE EVIDENCE, never on absence. A
non-anchored row is deleted only when a reconcile that succeeded at
observing did not name the node, and only when the node registry currently
reports that node reachable. Not “last_observed_at is old”: the thing
that makes a timestamp old is precisely the partition during which removal
must stay refused. The consequence is the invariant that makes this safe:
Stale or absent observation can only ever ADD refusals. It can never remove one.
That is the direct answer to “what happens when observation is stale or absent”, and it is the lesson from the follower-mTLS P1 restated as a data rule: a check that discharges itself when it cannot see is not a check.
Observation does not depend on the leader. Worth stating because it looks
like it does: swarm_recorder.go is leader-only and documents that rqlite
leadership normally parks on a swarm worker, where docker service ps
cannot answer, so that sweep stalls on both estates’ topology. But
container_recorder_node.go:450-462 gates on the shc.* label set, not
the compose-project label, and swarm task containers carry those labels, so
every node writes container.node_id for the swarm tasks running on it,
first-hand, every 30s floor pass. The set therefore converges without the
leader sweep, and a stalled sweep degrades to “rows held”, not “rows wrong”.
What the gate counts, and it is not one count
Section titled “What the gate counts, and it is not one count”Two readers, two different filters over the same table. Conflating them is how the drain hangs.
EnsureNodeUnoccupied(removal) counts ALL rows for the node: anchored OR observed. Either is a reason to refuse: anchored means bytes are here, observed means tasks are here. Still fails closed on an unwired lister.- Drain’s termination wait counts ANCHORED rows only.
shc modules/cluster/drain.go:600-619waits untilListByNodereturns zero. A drain discharges what SHC put on the node; a swarm task the scheduler still runs there is swarm’s business, and step 3 of the drain has already flipped availability todraining. Re-pointing this wait at the anchored subset is a REQUIRED co-change, not an optional one; against the full count a drain would never terminate. Counting anchored rows also means a half-finished move (both source and target anchored) cannot look drained.
The refusal gets better, not just correct
Section titled “The refusal gets better, not just correct”OccupancyBasis can delete its “SHC did not establish where the tasks or data
are now” clause, because now it did:
- anchored and observed → “its deployment directory is on this node and SHC has seen its containers here”
- anchored only → “SHC materialized it on this node (deployment directory, rendered compose), and has observed no containers here”
- observed only → “SHC has seen its containers running here; it was
materialized on
<anchor>”
One residual hole, named rather than hidden
Section titled “One residual hole, named rather than hidden”A stopped deployment whose local volume lives on a node that is not its
anchor (swarm + local volumes) loses its observed row on that node when the
containers go, and the volume is left with no occupancy row. That is a
volume occupancy question, not a deployment one, and it already has a
home: volume_attachment.current_node_id. The fix is a companion clause in
the same gate reading that table. Out of scope here, and it must not be
mistaken for something this table covers.
5. recorded_node_id: the reader inventory, and the verdict
Section titled “5. recorded_node_id: the reader inventory, and the verdict”VERDICT: IT STAYS. Narrowed to ROUTING, documented, no second rename.
The owner’s instinct is right that the gate does not need it and right that the set belongs in a join table. It is wrong that nothing else needs it. Enumerated across both repos, the readers fall into two groups, and the second group is not empty.
Moves to deployment_node
Section titled “Moves to deployment_node”| reader | file |
|---|---|
| the occupancy gate | ce modules/cluster/occupancy.go, via ListByNode |
| the three removal routes that discharge it | shc modules/cluster/admin_router.go:88,:239, lifecycle_router.go:347 |
| drain listing + termination wait | shc modules/cluster/drain.go:179,:494,:619 |
StackResponse.Nodes (already nodes on the wire) | ce modules/app/observed_nodes_enrich.go: one batched read replaces today’s per-row N+1 at router.go:397 |
| the fabric policy input | ce modules/fabric/policyinput.go:405-413, whose own comment asks for the observed set |
Stays on recorded_node_id: this is ROUTING, and routing is a scalar
Section titled “Stays on recorded_node_id: this is ROUTING, and routing is a scalar”| reader | file | why a set cannot answer |
|---|---|---|
| stack-verb target resolution | ce daemon/services_seams_stack.go:174,:195, services_stack_peer_resolve.go:96,:106, services_extra.go:1290 | StackTargetDescriptor.NodeID picks one daemon to execute on |
| backup capture target | ce daemon/backup_seam_adapters.go:267 | one node holds the deployment directory to tar |
AmITraefikHost | ce modules/app/deployment_repo.go GetDeploymentLocation | one host, by construction |
| health snapshot attribution | ce modules/health/{service.go:224,derive.go:166,deployment_poller.go:365,reconcile/reconcile.go:213,commands/main.go:233} | per-deployment scalar in the snapshot shape |
| collector labels | ce subsystems/impl/collectors.go:1057 | one label value |
| presumed-dead move plan | ce daemon/clone_dead_move_adapters.go:194 | the dead source is one node |
| compose self-heal | ce subsystems/impl/container_recorder_node.go:540-549 | rewrites the scalar for host-bound rows |
| host-bound render fallback | ce core/placement/placement.go Of | already guarded to host-bound only |
| the force sweep | ce modules/app/deployment_repo.go DeleteByNode | plus the DB-level RESTRICT backstop |
| the wire contract | ce StackResponse.NodeID json:"node"; shc pkg/apiclient/types_gen.go:2783 | published field |
| the HUD node column | shc modules/hud/adapters/deployments.go:200 → builder.go:601 | (should render placement.Format; separate lane) |
| DDL mirrors | shc tests/unit/subsystems/impl/stack_reconcile_ports_test.go:51, proxy_reconcile_cluster_test.go:32, tests/unit/modules/doctor/diagnoses/database_test.go | hand-written mirror of ce’s ent schema |
| docs | shc docs/reference/legacy-and-breaking.md:165,:339, docs/reference/api/response-models/payload-shapes.md | published reference |
“Which single node handled the install / holds the directory” is a real
question with a real, singular answer, asked on the hot path of every stack
verb. Deriving it from anchored = 1 would mean deriving a scalar from a
set that is transiently not a singleton (mid-move), the exact category
error this whole exercise exists to remove, reproduced one layer inward. It
would also discard the NOT NULL + RESTRICT FK, which is a genuine
integrity guarantee: no node row can vanish from under a deployment.
And it is unreachable anyway. HANDOFF-NEXT.md §37.2 records the
verification: ALTER TABLE DROP COLUMN refuses a column used in a foreign
key, and it cannot be made nullable; both require a rebuild, which is
forbidden. Since migrate/flatten is retired by owner ruling, “we will drop
it eventually” means never. So “keep it, narrowed, documented” is not a
compromise position; it is the only reachable state, and it happens to be the
correct one.
No second rename. RENAME COLUMN is FK-safe and free (d65b64b proved
it), but renaming the same column twice in two releases churns every mirror,
the wire and the docs for zero behaviour change. Narrow it in the field
comment and in §37.2’s table. It is not “provenance” any more; it is the
home node, and the ent field comment should say that its consumers are
routing and nothing else.
§37.2’s owed ruling is hereby answerable without the rebuild: the two-column shape is the end state. The staged full-database rebuild is not worth writing, because the thing it was going to buy (reclaiming the column) is not needed once the gate no longer stands on it.
6. Migration shape: one additive file, no rebuild
Section titled “6. Migration shape: one additive file, no rebuild”One new migration. recorded_node_id is not touched: no drop, no
nullability change, no second rename. Zero rebuild risk by construction.
The CREATE TABLE + two CREATE INDEX come straight from
make migrate NAME=deployment_node_set (verbatim as quoted in §2) because
for a new entity Atlas emits creation statements only. The backfill
Atlas cannot express, so it is hand-appended to the same file, exactly as the
pinned_node_id carry-forward UPDATE was:
-- ★ THE BACKFILL. Atlas generates the CREATE TABLE above and nothing else;-- these three statements are hand-written because the desired state cannot-- express "seed the set from the rows that already exist", and an empty-- occupancy set on the first boot after this migration would be a gate that-- permits every node removal on the estate.---- Anchor rows first: one per existing deployment, naming the node the old-- column recorded as holding its bytes. anchored=1 means SHC wrote to that-- node's filesystem, so observation may never prune it.INSERT INTO `deployment_node` (`deployment_id`, `node_id`, `anchored`, `first_seen_at`, `last_observed_at`)SELECT `id`, `recorded_node_id`, 1, `created_at`, NULL FROM `deployment` WHERE trim(coalesce(`recorded_node_id`, '')) <> '';-- Then observation, from the container attributions that already exist, so a-- swarm deployment carries its real node set from the first read rather than-- waiting for a reconcile pass. OR IGNORE because a node that is both the-- anchor and observed already has its row.INSERT OR IGNORE INTO `deployment_node` (`deployment_id`, `node_id`, `anchored`, `first_seen_at`, `last_observed_at`)SELECT `s`.`deployment_id`, `c`.`node_id`, 0, min(`c`.`created_at`), max(`c`.`updated_at`) FROM `container` `c` JOIN `service` `s` ON `c`.`service_id` = `s`.`id` WHERE trim(coalesce(`c`.`node_id`, '')) <> '' GROUP BY `s`.`deployment_id`, `c`.`node_id`;-- Finally stamp last_observed_at on the anchor rows that OR IGNORE just-- skipped, so an anchored-and-observed node is not recorded as never-- observed. Absence of a timestamp must mean "never seen", not "seen but the-- backfill lost it".UPDATE `deployment_node` SET `last_observed_at` = ( SELECT max(`c`.`updated_at`) FROM `container` `c` JOIN `service` `s` ON `c`.`service_id` = `s`.`id` WHERE `s`.`deployment_id` = `deployment_node`.`deployment_id` AND `c`.`node_id` = `deployment_node`.`node_id`) WHERE `last_observed_at` IS NULL AND EXISTS ( SELECT 1 FROM `container` `c` JOIN `service` `s` ON `c`.`service_id` = `s`.`id` WHERE `s`.`deployment_id` = `deployment_node`.`deployment_id` AND `c`.`node_id` = `deployment_node`.`node_id`);The note about the rebuild stays, both in d65b64b’s file and in a
pointer from this one; a future editor reaching for DROP COLUMN or a
nullability relaxation on deployment must land on the measurement, not
rediscover it.
Blast radius on the live estates: all 24 deployments across both are
compose (second-hand, from earlier read-only audits), so each takes one anchor
row naming the node it already names, plus observed rows for the containers
already attributed to that same node, i.e. one row per deployment with
anchored = 1 and last_observed_at set. Nothing moves.
FK directions, and why each way round:
deployment_id → deploymentCASCADE. Uninstall must clear the set, and it is the same cascade every other child ofdeploymentalready uses.node_id → nodeRESTRICT. The node side must NOT cascade. WithCASCADE, deleting a node row would delete the very rows that were the evidence against deleting it: a node removal silently discharging its own gate.RESTRICTpreserves the DB-level backstopDeploymentRepo.DeleteByNodealready documents, and the--forcepath still works: it deletes the deployments first, whose CASCADE clears the join rows, after which the node delete proceeds.
7. Proof, not assertion
Section titled “7. Proof, not assertion”A two-phase test in the shape of migrations/deployment_pinned_node_test.go:
catalogue applied up to the new file, real rows seeded at the pre-migration
schema with foreign_keys(1) on, then the new file applied. Written and run
green in a scratch ce worktree; both existing catalogue guards pass alongside
it (go test ./migrations/... ./core/db/... → ok).
Fixture: three nodes; a compose deployment on node-a with a container on
node-a; a swarm deployment whose install was handled on node-a and whose
tasks are on node-b and node-c, the exact false-negative shape from §1;
and a stopped swarm deployment on node-b with no containers at all.
Asserted and observed:
1/node-a anchored observed compose: anchor and observation agree2/node-a anchored unobserved swarm: install node, holds the directory, no tasks2/node-b free observed swarm: LIVE TASKS; today's gate returns 0 rows here2/node-c free observed swarm: LIVE TASKS; today's gate returns 0 rows here3/node-b anchored unobserved stopped: the anchor is the whole floorPlus, all four green:
- no child rows lost:
deployment,service,containercounts identical before and after; - FK enforcement still bites: an insert naming a nonexistent node is refused;
- the node side RESTRICTs:
DELETE FROM node WHERE id='node-b'is refused while the set names it; - the deployment side CASCADEs: deleting deployment 2 left zero occupancy rows.
Rows 2/node-b and 2/node-c are the defect, reproduced as data: today
ListByNode('node-b') returns zero and shc cluster rm node-b proceeds to
docker node rm --force.
What must also be tested when this is built, because each is a way the gate could pass while protecting nothing:
- the stopped deployment: containers reaped,
ObservedNodesempty, and the gate still refuses. This is the case a derived-only set gets wrong. - the unreachable node: the reconcile must NOT prune its rows. Assert the post-reconcile row count, not the reconcile’s return value.
- drain terminates: with the wait re-pointed at anchored rows, and a mutation of that filter back to the full count must make the test hang or fail. A drain test that passes against both filters is testing nothing.
- the false negative, end to end:
EnsureNodeUnoccupiedon the task-carrying non-install node must refuse. Verify this red againstorigin/mainfirst.
8. Verdict, in one paragraph
Section titled “8. Verdict, in one paragraph”The owner is right. The gate is unsound under swarm in the data-loss
direction, the rebuild measurement never applied to a new table, and
deployment_node is the only shape that is simultaneously always-non-empty
(anchor) and correct under a scheduler (observation), which neither shipped
column can be. pinned_node_id is already right and is not touched.
recorded_node_id cannot be dropped and should not be: routing is a genuine
scalar question with roughly a dozen live consumers, it carries a NOT NULL
RESTRICTguarantee a derived answer would lose, and the physical drop is unreachable without a forbidden rebuild. It stays, narrowed to routing, with a note saying why; a documented stranded column is better than a rebuild that eats child rows, and this one is not even stranded.