Skip to content
SHC Docs

The occupancy gate stands on the wrong column, and a join table is the fix

Superseded by New-line runtime. The text below is the record as of its date.

Status: design settled, ready to build: BUILT AND SHIPPING IN v0.0.19. ce 397823f implements this document as written: the deployment_node set, the new OccupancyLister port the gate reads, the four-times floor, the positive-evidence prune rule, the single-valued anchor, the fail-closed refusal with no fallback to DeploymentLister, and the additive migration with the node_id IN (SELECT id FROM node) filter §6 called load-bearing. recorded_node_id stayed, narrowed to routing, exactly as §5 ruled. The v0.0.19 pin bump carries it, so it changes estate behaviour from the next roll onward.

shc’s half of the landing is the test surface, in the same release: the route-level guards moved onto the occupancy port, the fail-closed matrix stopped being discharged by one unwired port, and two assertions were added that this repo did not have: that the gate refuses a node whose containers are observed here while the anchor listing answers zero (the false negative §1 names as the data-loss direction), and that it does NOT refuse on the anchor listing alone.

Verdict on the owner’s question, unchanged: he is right about the gate and right about the join table; he is wrong about recorded_node_id, which has a real surviving consumer and cannot go.

Baselines. ce origin/main 852c768, shc origin/main b4aed8e42. The shipped design under review is ce d65b64b, in v0.0.18. Every figure below was measured against those trees. No estate, cloud, DNS or pipeline was touched.


1. The verified defect: the gate is unsound in BOTH directions, and the

Section titled “1. The verified defect: the gate is unsound in BOTH directions, and the”

v0.0.18 split deployment.node_id into recorded_node_id (the rename; NOT NULL, RESTRICT FK to node) and pinned_node_id (new, nullable, FK-less, placement intent). The node-removal occupancy gate was deliberately left standing on recorded_node_id, “untouched”. That is the defect.

The chain, read end to end:

  1. What gets written. modules/stack/service.go recordStackRow: Node: firstNonEmpty(req.NodeID, target.NodeID). Under swarm with no --node, req.NodeID is empty, so the value stored is target.NodeID, the node that handled the install request.
  2. Where the tasks actually go. An unpinned swarm install stamps no placement constraint at all. install_runner.go:1169-1200 hands StampPlacement Node: opts.PlacementNode (the operator’s --node as a node name), and swarm_placement_replay.go is explicit that the recorded node is “deliberately NOT an input here” and that “an unpinned swarm deployment stays unpinned”. So swarm schedules the tasks on any eligible node.
  3. What the gate counts. modules/app/deployment_repo.go:112-123: SELECT … FROM deployment WHERE deployment.recorded_node_id = ?. cluster.Service.DeploymentsOnNode → EnsureNodeUnoccupied → X400017NODDEP.

Therefore, for a swarm deployment installed from node A with tasks scheduled on B and C:

removingrows returnedgatetruth
A1REFUSESno tasks here; only the deployment directory
B0PERMITSlive tasks and their local volumes are here

The false positive on A is an annoyance. The false negative on B is the data-loss direction, and the removal is not a paper one: modules/cluster/swarm_removal.go demotes-then-runs docker node rm --force (swarm_managers.go:491), so shc cluster rm B tears a live worker out of the swarm while SHC reports it had nothing on it. Any local-volume service whose task was running there is orphaned.

The worst instance is the automatic path. All three “remove another node” routes discharge this one gate (shc modules/cluster/admin_router.go:88 for the cluster repair sweep, :239 for cluster rm, lifecycle_router.go:347 for forget); the routes are EE, so ce’s own occupancy.go has no ce-side caller and grepping ce alone makes the gate look dead. cluster rm --force skips the gate by design and keeps doing so; the sweep and forget cannot. The repair sweep runs without an operator in the loop, and its own comment says so: “this is the AUTOMATIC removal path, so a 24-hour partition evicted a node carrying live stacks with no operator in the loop at all.” Under swarm it still does, for every node that is not the install node.

The code already admits the true positive is unsound too. modules/cluster/occupancy.go OccupancyBasis renders, for swarm rows:

“under swarm, where placement is swarm’s; this node is only what SHC recorded at install time, and SHC did not establish where the tasks or data are now”

A gate whose own refusal text says it established nothing about placement is not a placement gate. v0.0.18 made the rendering honest and left the decision wrong.


2. The rebuild argument does not reach a join table: MEASURED

Section titled “2. The rebuild argument does not reach a join table: MEASURED”

d65b64b’s hand-written migration is correct and its note must stay. Atlas generates a table rebuild for the deployment column change, and against rqlited v10.2.7 started with -fk (as core/rqlited does) with statements submitted as one transaction (as core/db.applyOne does) that rebuild measured deployment 3→1, service 1→0, host_mapping 1→0: DROP TABLE performs an implicit DELETE when foreign keys are enforced, so every ON DELETE CASCADE child is destroyed.

That measurement justifies the handling of the COLUMN change. It says nothing about a new table, and extending it to one was the error in the shipped design’s reasoning. Measured, not reasoned about; the join-table ent schema of §3 was scaffolded into a fresh ce worktree and make migrate run against it. Atlas emitted exactly this, and nothing else:

-- Create "deployment_node" table
CREATE TABLE `deployment_node` (
`id` integer NOT NULL PRIMARY KEY AUTOINCREMENT,
`anchored` bool NOT NULL DEFAULT false,
`first_seen_at` datetime NOT NULL,
`last_observed_at` datetime NULL,
`deployment_id` integer NOT NULL,
`node_id` text NOT NULL,
CONSTRAINT `deployment_node_node_occupied_by` FOREIGN KEY (`node_id`) REFERENCES `node` (`id`) ON UPDATE NO ACTION ON DELETE RESTRICT,
CONSTRAINT `deployment_node_deployment_occupancy` FOREIGN KEY (`deployment_id`) REFERENCES `deployment` (`id`) ON UPDATE NO ACTION ON DELETE CASCADE
);
-- Create index "deploymentnode_deployment_id_node_id" to table: "deployment_node"
CREATE UNIQUE INDEX `deploymentnode_deployment_id_node_id` ON `deployment_node` (`deployment_id`, `node_id`);
-- Create index "deploymentnode_node_id" to table: "deployment_node"
CREATE INDEX `deploymentnode_node_id` ON `deployment_node` (`node_id`);

No PRAGMA. No DROP TABLE. The deployment and node tables are not emitted at all; adding the inverse edges on their side moves no bytes, because the FK columns live on the child. migrations/catalogue_shape_test.go and core/db/rebuild_guard_test.go both pass on the file as generated.

There is precedent in the schema for exactly this shape: ent/schema/store_node.go is already a (store_id, node_id) join table with two FK edges, and 20260803170107_connection_tombstone.sql is an Atlas-generated additive CREATE TABLE in the live catalogue.


Three facts, three homes. The shipped design has two of them right.

factquestionhomeshape
INTENTwhere did the operator ask for it?deployment.pinned_node_idscalar, nullable, no FK; unchanged, already correct
OCCUPANCYwhich nodes hold its bytes or run its tasks?deployment_node (new)a SET, always non-empty for a live deployment
ROUTINGwhich single daemon executes its verbs?deployment.recorded_node_idscalar, NOT NULL, RESTRICT FK; kept, narrowed (§5)
ent/schema/deployment_node.go
field.Int("deployment_id") // → deployment, CASCADE
field.String("node_id").MaxLen(255) // → node, RESTRICT
field.Bool("anchored").Default(false) // SHC wrote bytes to this node's disk
field.Time("first_seen_at")
field.Time("last_observed_at").Optional().Nillable() // NULL = never observed
// index.Fields("deployment_id", "node_id").Unique(); index.Fields("node_id")
// edge.From("deployment", …).Ref("occupancy").Field("deployment_id").Unique().Required()
// edge.From("node", …).Ref("occupied_by").Field("node_id").Unique().Required()

Two independent facts per row, not one enum. anchored means SHC put bytes on this node’s filesystem: the rendered compose, the state manifest, the deployment directory. last_observed_at means SHC has seen containers of this deployment here. A node can be either, both, or (transiently) neither. Collapsing them into one origin column forces a lossy choice for the node that is both, which is the ordinary compose case.

This is the thing neither shipped column can be. It is always non-empty for a live deployment, because the anchor is written by the same install path that writes recorded_node_id, so the gate keeps its floor. And it is correct under swarm, because observation adds and removes rows. recorded_node_id has the floor without the correctness; a derived-only set has the correctness without the floor.

Derived-only is not survivable, and the reason is in the tree. The observed set legitimately reads EMPTY for a running or restorable deployment in at least three shapes:

  • A stopped deployment. shc stop removes the containers; the leader reaper (container_reconciler.go, updated_at < cutoff, 120s) deletes the rows. ObservedNodes then returns nil while the compose deployment directory and every local volume still sit on that node’s disk.
  • An unreachable node. Its recorder stops heartbeating (TouchContainers), the leader reaps its rows after 120s, and the derived set empties, during the exact outage that makes removal dangerous. This is the cluster repair sweep’s whole subject class.
  • The window before first observation. A fresh install has no container rows for up to one recorder pass.

A gate that derives to zero in those three cases is a gate that permits removing a node that holds live state. The anchor row is what makes the answer never zero.

pinned_node_id stays exactly as it is. Intent is a different question from occupancy, its FK-lessness is deliberate (a pin naming a removed node is information, not corruption), and app.PlacementIntent already owns the one rule that fills it. Nothing here touches it.


Four writers, and only four.

  1. Install / re-render: the recordStackRow → UpsertStack path. UPSERT the row for the resolved node with anchored = 1. Idempotent. Written from the same call that writes recorded_node_id, so the two cannot disagree about which node materialized the deployment.
  2. Observation: a leader-only occupancy reconcile, folded into the existing ContainerReconciler (60s, already leader-only, already authorized to delete container rows). Per deployment: UPSERT a row for every node the container-derived set names, stamping last_observed_at; then DELETE non-anchored rows the derivation does not name.
  3. Move / drain: app.DeploymentRepo.Migrate. Set anchored = 0 on the old node’s row (leaving last_observed_at alone), UPSERT the target’s row with anchored = 1. The pin rule already there is unchanged: a move follows an existing pin and never creates one.
  4. Uninstall: nothing to write. ON DELETE CASCADE from deployment clears the set. MEASURED (§7): deleting a deployment left zero occupancy rows.

★ THE PRUNE RULE: prune on POSITIVE EVIDENCE, never on absence. A non-anchored row is deleted only when a reconcile that succeeded at observing did not name the node, and only when the node registry currently reports that node reachable. Not “last_observed_at is old”: the thing that makes a timestamp old is precisely the partition during which removal must stay refused. The consequence is the invariant that makes this safe:

Stale or absent observation can only ever ADD refusals. It can never remove one.

That is the direct answer to “what happens when observation is stale or absent”, and it is the lesson from the follower-mTLS P1 restated as a data rule: a check that discharges itself when it cannot see is not a check.

Observation does not depend on the leader. Worth stating because it looks like it does: swarm_recorder.go is leader-only and documents that rqlite leadership normally parks on a swarm worker, where docker service ps cannot answer, so that sweep stalls on both estates’ topology. But container_recorder_node.go:450-462 gates on the shc.* label set, not the compose-project label, and swarm task containers carry those labels, so every node writes container.node_id for the swarm tasks running on it, first-hand, every 30s floor pass. The set therefore converges without the leader sweep, and a stalled sweep degrades to “rows held”, not “rows wrong”.

What the gate counts, and it is not one count

Section titled “What the gate counts, and it is not one count”

Two readers, two different filters over the same table. Conflating them is how the drain hangs.

  • EnsureNodeUnoccupied (removal) counts ALL rows for the node: anchored OR observed. Either is a reason to refuse: anchored means bytes are here, observed means tasks are here. Still fails closed on an unwired lister.
  • Drain’s termination wait counts ANCHORED rows only. shc modules/cluster/drain.go:600-619 waits until ListByNode returns zero. A drain discharges what SHC put on the node; a swarm task the scheduler still runs there is swarm’s business, and step 3 of the drain has already flipped availability to draining. Re-pointing this wait at the anchored subset is a REQUIRED co-change, not an optional one; against the full count a drain would never terminate. Counting anchored rows also means a half-finished move (both source and target anchored) cannot look drained.

OccupancyBasis can delete its “SHC did not establish where the tasks or data are now” clause, because now it did:

  • anchored and observed → “its deployment directory is on this node and SHC has seen its containers here”
  • anchored only → “SHC materialized it on this node (deployment directory, rendered compose), and has observed no containers here”
  • observed only → “SHC has seen its containers running here; it was materialized on <anchor>”

One residual hole, named rather than hidden

Section titled “One residual hole, named rather than hidden”

A stopped deployment whose local volume lives on a node that is not its anchor (swarm + local volumes) loses its observed row on that node when the containers go, and the volume is left with no occupancy row. That is a volume occupancy question, not a deployment one, and it already has a home: volume_attachment.current_node_id. The fix is a companion clause in the same gate reading that table. Out of scope here, and it must not be mistaken for something this table covers.


5. recorded_node_id: the reader inventory, and the verdict

Section titled “5. recorded_node_id: the reader inventory, and the verdict”

VERDICT: IT STAYS. Narrowed to ROUTING, documented, no second rename.

The owner’s instinct is right that the gate does not need it and right that the set belongs in a join table. It is wrong that nothing else needs it. Enumerated across both repos, the readers fall into two groups, and the second group is not empty.

readerfile
the occupancy gatece modules/cluster/occupancy.go, via ListByNode
the three removal routes that discharge itshc modules/cluster/admin_router.go:88,:239, lifecycle_router.go:347
drain listing + termination waitshc modules/cluster/drain.go:179,:494,:619
StackResponse.Nodes (already nodes on the wire)ce modules/app/observed_nodes_enrich.go: one batched read replaces today’s per-row N+1 at router.go:397
the fabric policy inputce modules/fabric/policyinput.go:405-413, whose own comment asks for the observed set

Stays on recorded_node_id: this is ROUTING, and routing is a scalar

Section titled “Stays on recorded_node_id: this is ROUTING, and routing is a scalar”
readerfilewhy a set cannot answer
stack-verb target resolutionce daemon/services_seams_stack.go:174,:195, services_stack_peer_resolve.go:96,:106, services_extra.go:1290StackTargetDescriptor.NodeID picks one daemon to execute on
backup capture targetce daemon/backup_seam_adapters.go:267one node holds the deployment directory to tar
AmITraefikHostce modules/app/deployment_repo.go GetDeploymentLocationone host, by construction
health snapshot attributionce modules/health/{service.go:224,derive.go:166,deployment_poller.go:365,reconcile/reconcile.go:213,commands/main.go:233}per-deployment scalar in the snapshot shape
collector labelsce subsystems/impl/collectors.go:1057one label value
presumed-dead move plance daemon/clone_dead_move_adapters.go:194the dead source is one node
compose self-healce subsystems/impl/container_recorder_node.go:540-549rewrites the scalar for host-bound rows
host-bound render fallbackce core/placement/placement.go Ofalready guarded to host-bound only
the force sweepce modules/app/deployment_repo.go DeleteByNodeplus the DB-level RESTRICT backstop
the wire contractce StackResponse.NodeID json:"node"; shc pkg/apiclient/types_gen.go:2783published field
the HUD node columnshc modules/hud/adapters/deployments.go:200 → builder.go:601(should render placement.Format; separate lane)
DDL mirrorsshc tests/unit/subsystems/impl/stack_reconcile_ports_test.go:51, proxy_reconcile_cluster_test.go:32, tests/unit/modules/doctor/diagnoses/database_test.gohand-written mirror of ce’s ent schema
docsshc docs/reference/legacy-and-breaking.md:165,:339, docs/reference/api/response-models/payload-shapes.mdpublished reference

“Which single node handled the install / holds the directory” is a real question with a real, singular answer, asked on the hot path of every stack verb. Deriving it from anchored = 1 would mean deriving a scalar from a set that is transiently not a singleton (mid-move), the exact category error this whole exercise exists to remove, reproduced one layer inward. It would also discard the NOT NULL + RESTRICT FK, which is a genuine integrity guarantee: no node row can vanish from under a deployment.

And it is unreachable anyway. HANDOFF-NEXT.md §37.2 records the verification: ALTER TABLE DROP COLUMN refuses a column used in a foreign key, and it cannot be made nullable; both require a rebuild, which is forbidden. Since migrate/flatten is retired by owner ruling, “we will drop it eventually” means never. So “keep it, narrowed, documented” is not a compromise position; it is the only reachable state, and it happens to be the correct one.

No second rename. RENAME COLUMN is FK-safe and free (d65b64b proved it), but renaming the same column twice in two releases churns every mirror, the wire and the docs for zero behaviour change. Narrow it in the field comment and in §37.2’s table. It is not “provenance” any more; it is the home node, and the ent field comment should say that its consumers are routing and nothing else.

§37.2’s owed ruling is hereby answerable without the rebuild: the two-column shape is the end state. The staged full-database rebuild is not worth writing, because the thing it was going to buy (reclaiming the column) is not needed once the gate no longer stands on it.


6. Migration shape: one additive file, no rebuild

Section titled “6. Migration shape: one additive file, no rebuild”

One new migration. recorded_node_id is not touched: no drop, no nullability change, no second rename. Zero rebuild risk by construction.

The CREATE TABLE + two CREATE INDEX come straight from make migrate NAME=deployment_node_set (verbatim as quoted in §2) because for a new entity Atlas emits creation statements only. The backfill Atlas cannot express, so it is hand-appended to the same file, exactly as the pinned_node_id carry-forward UPDATE was:

-- ★ THE BACKFILL. Atlas generates the CREATE TABLE above and nothing else;
-- these three statements are hand-written because the desired state cannot
-- express "seed the set from the rows that already exist", and an empty
-- occupancy set on the first boot after this migration would be a gate that
-- permits every node removal on the estate.
--
-- Anchor rows first: one per existing deployment, naming the node the old
-- column recorded as holding its bytes. anchored=1 means SHC wrote to that
-- node's filesystem, so observation may never prune it.
INSERT INTO `deployment_node`
(`deployment_id`, `node_id`, `anchored`, `first_seen_at`, `last_observed_at`)
SELECT `id`, `recorded_node_id`, 1, `created_at`, NULL
FROM `deployment`
WHERE trim(coalesce(`recorded_node_id`, '')) <> '';
-- Then observation, from the container attributions that already exist, so a
-- swarm deployment carries its real node set from the first read rather than
-- waiting for a reconcile pass. OR IGNORE because a node that is both the
-- anchor and observed already has its row.
INSERT OR IGNORE INTO `deployment_node`
(`deployment_id`, `node_id`, `anchored`, `first_seen_at`, `last_observed_at`)
SELECT `s`.`deployment_id`, `c`.`node_id`, 0, min(`c`.`created_at`), max(`c`.`updated_at`)
FROM `container` `c`
JOIN `service` `s` ON `c`.`service_id` = `s`.`id`
WHERE trim(coalesce(`c`.`node_id`, '')) <> ''
GROUP BY `s`.`deployment_id`, `c`.`node_id`;
-- Finally stamp last_observed_at on the anchor rows that OR IGNORE just
-- skipped, so an anchored-and-observed node is not recorded as never
-- observed. Absence of a timestamp must mean "never seen", not "seen but the
-- backfill lost it".
UPDATE `deployment_node`
SET `last_observed_at` = (
SELECT max(`c`.`updated_at`)
FROM `container` `c`
JOIN `service` `s` ON `c`.`service_id` = `s`.`id`
WHERE `s`.`deployment_id` = `deployment_node`.`deployment_id`
AND `c`.`node_id` = `deployment_node`.`node_id`)
WHERE `last_observed_at` IS NULL
AND EXISTS (
SELECT 1
FROM `container` `c`
JOIN `service` `s` ON `c`.`service_id` = `s`.`id`
WHERE `s`.`deployment_id` = `deployment_node`.`deployment_id`
AND `c`.`node_id` = `deployment_node`.`node_id`);

The note about the rebuild stays, both in d65b64b’s file and in a pointer from this one; a future editor reaching for DROP COLUMN or a nullability relaxation on deployment must land on the measurement, not rediscover it.

Blast radius on the live estates: all 24 deployments across both are compose (second-hand, from earlier read-only audits), so each takes one anchor row naming the node it already names, plus observed rows for the containers already attributed to that same node, i.e. one row per deployment with anchored = 1 and last_observed_at set. Nothing moves.

FK directions, and why each way round:

  • deployment_id → deployment CASCADE. Uninstall must clear the set, and it is the same cascade every other child of deployment already uses.
  • node_id → node RESTRICT. The node side must NOT cascade. With CASCADE, deleting a node row would delete the very rows that were the evidence against deleting it: a node removal silently discharging its own gate. RESTRICT preserves the DB-level backstop DeploymentRepo.DeleteByNode already documents, and the --force path still works: it deletes the deployments first, whose CASCADE clears the join rows, after which the node delete proceeds.

A two-phase test in the shape of migrations/deployment_pinned_node_test.go: catalogue applied up to the new file, real rows seeded at the pre-migration schema with foreign_keys(1) on, then the new file applied. Written and run green in a scratch ce worktree; both existing catalogue guards pass alongside it (go test ./migrations/... ./core/db/... → ok).

Fixture: three nodes; a compose deployment on node-a with a container on node-a; a swarm deployment whose install was handled on node-a and whose tasks are on node-b and node-c, the exact false-negative shape from §1; and a stopped swarm deployment on node-b with no containers at all.

Asserted and observed:

1/node-a anchored observed compose: anchor and observation agree
2/node-a anchored unobserved swarm: install node, holds the directory, no tasks
2/node-b free observed swarm: LIVE TASKS; today's gate returns 0 rows here
2/node-c free observed swarm: LIVE TASKS; today's gate returns 0 rows here
3/node-b anchored unobserved stopped: the anchor is the whole floor

Plus, all four green:

  • no child rows lost: deployment, service, container counts identical before and after;
  • FK enforcement still bites: an insert naming a nonexistent node is refused;
  • the node side RESTRICTs: DELETE FROM node WHERE id='node-b' is refused while the set names it;
  • the deployment side CASCADEs: deleting deployment 2 left zero occupancy rows.

Rows 2/node-b and 2/node-c are the defect, reproduced as data: today ListByNode('node-b') returns zero and shc cluster rm node-b proceeds to docker node rm --force.

What must also be tested when this is built, because each is a way the gate could pass while protecting nothing:

  • the stopped deployment: containers reaped, ObservedNodes empty, and the gate still refuses. This is the case a derived-only set gets wrong.
  • the unreachable node: the reconcile must NOT prune its rows. Assert the post-reconcile row count, not the reconcile’s return value.
  • drain terminates: with the wait re-pointed at anchored rows, and a mutation of that filter back to the full count must make the test hang or fail. A drain test that passes against both filters is testing nothing.
  • the false negative, end to end: EnsureNodeUnoccupied on the task-carrying non-install node must refuse. Verify this red against origin/main first.

The owner is right. The gate is unsound under swarm in the data-loss direction, the rebuild measurement never applied to a new table, and deployment_node is the only shape that is simultaneously always-non-empty (anchor) and correct under a scheduler (observation), which neither shipped column can be. pinned_node_id is already right and is not touched. recorded_node_id cannot be dropped and should not be: routing is a genuine scalar question with roughly a dozen live consumers, it carries a NOT NULL

  • RESTRICT guarantee a derived answer would lose, and the physical drop is unreachable without a forbidden rebuild. It stays, narrowed to routing, with a note saying why; a documented stranded column is better than a rebuild that eats child rows, and this one is not even stranded.