Skip to content
SHC Docs

One virtual IP per service (design)

Status: open design (SHC-1238). Allocation, the VIP table and the per-node applier are built, and so is the sidecar case (SHC-1298): every rule of section 1’s SHC-1298 paragraph has code and tests in ce (fabric_pin.members, SettleNamespaces, member backends in modules/vip/section.go, MemberReady, the yield rule on both the nft and the pf compilers, X400240PODNET, X400241PODPUB). Its live proof on the rig is still owed. Clay, 2026-10-02: an SHC service (one compose service and its replicas) always has exactly one virtual IP, like a Kubernetes Service. “Routing wise they all share a single virtual IP … I feel like that’s just how that should work.” The authored stateful flag that would have taken the VIP away is rejected. Groups, members and aliases on top of replicas are not ruled and are out of scope here, as are external load balancing (--port, --host) and the mesh.

A service web of deployment acme/prod/shop has one address, say 10.224.0.17. A container anywhere in the cluster that connects to 10.224.0.17:8080 lands on one ready replica of web, on port 8080 of that replica’s fabric pin. The address never changes while the service exists: not on update, not on restart, not when a replica moves.

The VIP exists only as a match in nftables. No interface carries it, no route points at it, and it never appears on the wire: the node the client runs on rewrites the destination to a replica’s pin in nat prerouting, and the packet crosses the fabric addressed to that pin. So the VIP needs no WireGuard AllowedIPs, no ARP and no route, and it works the same on one node or fifty.

Services of one network namespace (SHC-1298). A service that joins another’s network namespace (network_mode: service:web, a sidecar in the same pod) has no address of its own: it listens on its owner’s. It still has its own VIP. The owner’s pin row records it as a member with the container ports it declares (fabric_pin.members; a second row cannot carry the same address, because (node_id, ordinal) is unique), the VIP table gives the member’s VIP the owner’s pin as its backend, judged by the member’s own container, and the tenant policy opens the member’s declared ports on that address. The two names split the address by port: each VIP yields, and so refuses, the ports another service of its namespace declares and it does not, counted as port ranges (a port one names inside the other’s range is carried by both) and over every service of the namespace, with a VIP yet or not. A flow rewritten before a port changed hands ends on the next flow pass. A ready owner with a dead member takes no traffic on the member’s VIP, and a route to the member waits for the member’s own newest container, never the owner’s. A member that declares no port gets what any portless service gets. A member is never published or routed under its owner’s name: a --host route to the owner on a port only a member declares finds no candidate (X400936HSTNOP), a --port publish naming the owner for such a port, or naming a member at all, is refused (X400241PODPUB), and a member whose owner has no fabric address is refused, ports or none (X400240PODNET), since its VIP could have no backend. A port the owner declares itself stays the owner’s to publish and route. Each render settles the deployment’s pins against its namespaces, under a per-deployment lock on its node: a service that now joins another’s loses the pin it held as an owner, and a pin whose service left the document stops naming the document’s services as members (other names it holds, such as a replica entry’s, stay). Why a VIP of its own and not the owner’s: Clay ruled that a service always has exactly one VIP, and that a replica is a pod whose members move as one. Answering the member’s name with the owner’s VIP would leave a service with no VIP and make its name an alias of another service’s, and aliases are not ruled. A VIP over the owner’s pins keeps both rulings and asks nothing new of either, so the alias form goes to Clay before anyone builds it.

Range. 10.224.0.0/11, the upper half of the control-plane /10 ruled on 2026-10-01. shc init refuses a fabric CIDR that overlaps it (X400351FABCID). A cluster recorded with one before that keeps its fabric, and its posture is still adopted, but it runs no VIPs: nothing is allocated, the section’s table names no fabric, LiveVIPs answers nothing, the policy’s error accept stays pin-only (section 6), and the leader warns once (X409239VIPOVL) until the fabric moves to a disjoint supernet. Such a cluster, and one off the shc fabric, runs no replica projects either (SHC-1379): a replica’s peers and Traefik dial its service’s VIP, and nothing falls back to its pins. The daemon declines them (X200769PODOFF), naming the VIP module’s own refusal as the reason shc doctor shows: X409249VIPNSF off the shc fabric, X409250VIPOVR on an overlap, X409243VIPUNR for a posture that does not read, and X409251VIPNAP on a node whose platform runs no VIP applier (every darwin node until SHC-1381). A daemon that did decide to run them asks again before every pass that can start a replica (Runner.VIPs, over the posture as the switch reads it), and a refusal starts nothing; a stop, a removal and a prune do not ask. One predicate, compile.OverlapsDefaultRange, decides all of it, over one reading of the CIDR: fabric.ParseCIDR. Go’s two parsers disagree on some spellings (10.224.0.0/012 parses with net only), so Posture.Normalize rewrites the CIDR to its canonical masked form for every consumer, a CIDR nothing can read counts as overlapping, and a node refuses a table whose fabric overlaps the range before it loads a rule. The lower half, 10.192.0.0/11, stays with mesh addresses. That is 2,097,152 addresses; an address whose last octet is 0 or 255 is never handed out, because some client stacks refuse them.

Allocated on the leader, stored in rqlite. New table service_vip, in the single squashed migration:

columnmeaning
iduuid
deployment_keytenant/environment/stack, the pin ledger’s key
service_namethe compose service, never a replica entry (web, not web-r0)
service_idthe service row it was handed to; AUTOINCREMENT, never reused
deployment_idthat row’s deployment row; AUTOINCREMENT, never reused
slotoffset into the range; the address is range + slot
seqallocation order, one above the highest any row holds
addressthe rendered address, for readers that do not do the math
created_at, updated_at
released_atwhen the service went away, a UTC datetime like every store time; NULL while it lives

UNIQUE (slot) and UNIQUE (seq) over every row, live or released. UNIQUE (deployment_key, service_name) over live rows (released_at IS NULL). Same doctrine as fabric_pin: no interactive transaction, every write self-contained, a constraint or 0-rows outcome rereads and retries. Each write puts clusterfs.BumpStmt in the same batch, so the VIP and the revision that publishes it replicate together.

Who allocates. A leader-gated sweep, every 5 s:

  • every service row joined to its live deployment row that has no live VIP gets one;
  • every live VIP whose (deployment, service, service_id, deployment_id) no longer joins is released: released_at is stamped and the row stays. A service removed and added again under the same name has a new service id, a reinstall has a new deployment id, and a service row that RescueOrphanedServiceRow relinks to a reinstalled deployment has a new pair, so each gets a new VIP even when no sweep ran in between.

The sweep reads the live VIPs before the services: a service missing from the later read is gone for good, because its ids never come back. A release names the allocation by its seq, which no other allocation ever carries, and proves in the same WHERE that the lifetime it was handed to is gone (no service row with that id, name, deployment row and key). An allocation re-asserts the same identity for itself. So no writer, however stale its read, takes a VIP from a service that still exists. No order between lifetimes is assumed: a relinked row keeps an older service id under a newer deployment. One service that cannot get a VIP holds none of the others back.

The install and update path may call the same idempotent Ensure so a fresh service has its VIP before its first replica starts; the sweep is what makes it true regardless. A failed read releases nothing. A service row is upserted by (deployment_id, name), so an update never makes a service disappear for a moment and never costs it its VIP.

Order and reuse. No address is reused before the range is exhausted. The allocator is a cursor that only moves forward through slots no row has ever held: the newest row (highest seq) names the last slot handed out, and the next allocation takes the first never-held usable slot after it. Released rows stay as tombstones and are skipped, so for the first two million allocations every VIP is a fresh address.

Only when no never-held slot is left does the allocator reuse, and then it takes the tombstone released longest ago, provided that release is older than VIPQuarantine (24 hours); it deletes the tombstone and writes the new row in one batch. With no such tombstone the allocation is refused (range full). An address therefore comes back only after every other address in the range has been handed out once, oldest release first.

That makes the contract explicit. A VIP belongs to its service for as long as the service exists. A client that keeps an address after its service is uninstalled reaches nothing until the range is exhausted, and after that may reach whichever service holds it next; resolving the name again is how a client stays right, which the 5 s TTL is for. Every node that is converging current revisions ends tracked flows to a VIP no longer live in its table (section 6). A node partitioned from the store keeps routing from its last revision until its clusterfs catches up, which it shares with pins and routes; the oldest release first rule puts the most time possible between that node’s last view and any reuse.

Keyed on (deployment, service) and the service row, and on nothing about placement, the VIP survives an update, a restart and a relocation unchanged (AC1): an update upserts the service row by (deployment_id, name) and keeps its id. An uninstall releases it; a reinstall under the same name gets a new one.

VIPs are allocated only on a cluster whose fabric posture is shc. Without the fabric there are no pins to route to, the section’s table names no fabric, and every node’s applier leaves its host alone (section 7).

A new clusterfs section, vip, rendered on every node like resolver and traefik. It writes one file, vip/services.json:

{
"range": "10.224.0.0/11",
"fabric": "10.128.0.0/10",
"nodes": { "n1": { "location": "fsn" }, "n2": { "location": "fsn" }, "n3": { "location": "hel" } },
"services": [
{
"deployment": "acme/prod/shop",
"service": "web",
"vip": "10.224.0.17",
"backends": [
{ "replica": "web-r0", "pin": "10.128.16.130", "node": "n1", "running": true, "ready": true },
{ "replica": "web-r1", "pin": "10.128.32.130", "node": "n2", "running": true, "ready": false }
]
}
]
}
  • backends are the service’s live pins: the pin filed under web or under a replica entry web-r<N>, with the node it sits on.

  • ready needs positive evidence about that backend’s own replica, from a source that records it under the same key the pin is filed under:

    • a pin filed under the service name (web) is the one container of that service; pins are unique per service, so no other container can carry the address. Its evidence is the newest container row the node recorded for that deployment, service and node.
    • a pin filed under a replica entry (web-r0) is that replica’s, and container rows cannot name it: the recorder files a replica’s container under its base service and keeps no ordinal. The assignment store cannot yet either: its state is the pod replica’s, and a pod’s readiness is taken from the members that declare ports, which need not include web. So a replica-entry backend needs member-specific evidence, reported per member of each pod replica at the current generation, and until nodes report that (SHC-1144, over SHC-1216’s replica records) it is never ready. A replica-entry VIP carries no traffic before its evidence exists. Whoever wires that report bumps the clusterfs revision in the same batch as an accepted change of a member’s running or ready state, as the container recorder does, so the VIP republishes on the change and not at the next refresh.

    Ready means running, and healthy or without a healthcheck (the node agent’s damper has already held back a flapping health word). No evidence means not ready, so a new or replacement pin takes no traffic before its container is up, and there is no fallback from a replica entry to its service, so a healthy sibling never speaks for another replica. The node must also be ready and the deployment serving. This is stricter than the resolver’s predicate today, which reads an unobserved replica as ready and ORs a service’s containers on a node; the two should converge on this one.

  • running is whether that same evidence says the replica runs at all, whatever its health. It decides which flows survive (section 6). The container recorder bumps the clusterfs revision when a container’s running state changes as well as when its readiness does, so a running but unhealthy container that exits is republished at once, not at the next refresh.

  • Sorted throughout, so an unchanged table has an unchanged section digest and the node does nothing.

The file also carries the cluster’s published ports (SHC-1353), under published:

"published": [
{ "deployment": "acme/prod/shop", "service": "web", "port": 8443, "container": 443,
"protocol": "tcp", "nodes": ["n1", "n2"] }
]
  • One entry per published port, read from the ledger: the published_ports column of each deployment’s row. port is the port a listening node takes connections on and container the port the replicas listen on. spread is set for a port published with +spread.
  • nodes is the listen set, resolved on every build from the port’s selector (or the cluster default publish.listen, or every node), the node labels and the live pins. A label, a replica or a node that moves takes the listeners with it at the next build.
  • No two entries share a node, port and protocol. Install, update and shc node label refuse an overlap. Where one arises anyway (a replica moves, a location changes) the older deployment keeps the node and the younger listens on the rest of its set.
  • A port of a service with no live VIP is left out.

The section is cluster-wide content. Each node compiles its own ruleset from it (section 4), which is the only node-specific step.

The node applier reads current/vip/services.json, compiles it for itself and converges the kernel with the fabric’s existing discipline: one nft -f transaction (declare, delete, recreate the table), an applied marker chain carrying the document’s version, read back from the kernel. A marker that already matches is a strict no-op.

table ip shc_vip {
map dispatch {
type ipv4_addr : verdict
elements = { 10.224.0.17 : goto vip_10_224_0_17 }
}
set hairpin {
type ipv4_addr . ipv4_addr
elements = { 10.128.16.130 . 10.128.16.130 }
}
chain prerouting {
type nat hook prerouting priority dstnat; policy accept;
ip daddr vmap @dispatch
}
chain output {
type nat hook output priority dstnat; policy accept;
ip daddr vmap @dispatch
}
chain postrouting {
type nat hook postrouting priority srcnat; policy accept;
ct status dnat ip saddr . ip daddr @hairpin counter masquerade
ct status dnat ct original ip daddr 10.224.0.0/11 fib saddr type local counter masquerade
}
chain refuse_forward {
type filter hook forward priority -30; policy accept;
ip daddr 10.224.0.0/11 meta l4proto tcp counter reject with tcp reset
ip daddr 10.224.0.0/11 counter reject with icmp type port-unreachable
}
chain refuse_output { ... same two rules on the output hook ... }
chain vip_10_224_0_17 {
counter dnat ip to 10.128.16.130 comment "acme/prod/shop/web local"
}
chain applied { counter comment "applied sha256:..." }
}

Published ports. A node in a published port’s nodes gets three more rules in the same table:

chain prerouting {
type nat hook prerouting priority dstnat; policy accept;
iifname != { "shc0", "mesh0" } fib daddr type local jump published
ip daddr vmap @dispatch
}
chain postrouting {
...
oifname "shc0" ct status dnat ct mark & 0x10000000 == 0x10000000 counter masquerade
}
chain published {
tcp dport 8443 counter ct mark set ct mark or 0x10000000 dnat to 10.128.16.130:443 comment "acme/prod/shop/web 8443/tcp local"
}

A published port is a rule, never a socket and never a runtime’s own publish: the rendered compose has no ports: entry for it. A connection that arrives for any of the node’s own addresses on the port, from anywhere but the fabric device and the control-plane mesh, is rewritten to a ready replica’s pin on the container port. The target is where the service’s VIP goes from that node (nearest first, below), so a replica on the listening node takes it and sees the client’s own address. With +spread it is every ready replica. A flow that leaves by the fabric device wears the node’s fabric address, the only source a peer’s AllowedIPs admit from it.

The rule that rewrites also sets the conntrack mark the fabric policy reads (policy.PublicationCtMark). It is one rule, so the marked flows are exactly the flows the listener published, and nothing else in SHC sets the bit. The policy admits a client from outside the fabric to a pin on another node only with the rewrite and the mark (section 6). A client that dials the pin itself, or that another table rewrote to it, carries no mark and is dropped; the kernel lab measures both.

With no ready replica the node has no rule for the port, and its own stack refuses the connection. A node outside the set has none of the three rules. The table is IPv4, so a published port does not answer on a node’s IPv6 address, and it does not answer a connection the node makes to itself (nat prerouting never sees one).

An app’s own ports: entries are not publishes. On a cluster that runs VIPs the render takes them out and declares their container ports under expose:, and the install warns (X200141PUBTPL) with the --expose that would publish each. The system Traefik’s 80 and 443, and its SSH entrypoint when one is set, are the one exception: they become plain publishes with the listen selector node=@(service=traefik), so a node listens on them exactly while it holds a Traefik replica and always sends the connection to that replica, which sees the client’s address. A process on a node reaches Traefik through its service VIP (vip.LocalAddr): the keycloak and OTLP backchannels and the certificate probes dial it, since a node’s loopback no longer answers 443.

Address-only DNAT. dnat ip to <pin> rewrites the address and keeps the port, so the VIP is the service’s address on every port. Which ports a client may use is the tenant policy’s call (section 6), exactly as if it had dialled the pin. Inside a deployment everything talks; across deployments only declared ports pass.

Nearest first, per node. Of the service’s ready backends the node keeps the first non-empty tier: those on this node, else those under the deepest location level this node shares with any ready backend, else all of them. Levels compare whole path segments; an empty location shares no level. Several in the chosen tier are spread per connection with numgen random mod N map { 0 : pin0, 1 : pin1 }; conntrack pins each flow to its first choice. Every dispatch rule carries a counter and a comment naming the tier (local, location, any), which is what the kernel proof reads (AC5).

No ready backend means no dispatch element. The packet stays addressed to the VIP and refuse_forward answers at once with a TCP reset or ICMP port unreachable, the code a connected UDP socket fails on. The same rule catches a VIP nobody owns, and refuse_output answers a process in the node’s own namespace the same way, which also keeps its VIP traffic off the uplink.

Hairpin. A replica that dials its own service’s VIP and is picked gets a packet from itself to itself. Two things make that work, and Kubernetes needs the same two. First, br_netfilter treats a DNAT whose new destination sits on the bridge the packet came from as a bridged DNAT and hands the frame back to the bridge with the ingress port as its device; the bridge then delivers to the destination port only if that is a different port or the port is in hairpin_mode. So the applier turns hairpin_mode on for every port of every fabric bridge, and checks it again on every tick, because a recreated container comes back on a new veth. Second, without a source rewrite the replica’s reply would stay inside its own namespace and never be un-NATted, so postrouting masquerades exactly the pairs (pin . pin) of this node’s chosen local backends, after the forward verdict, and the reply comes back through the host.

Two different containers on the same bridge need neither: the frame goes to a different port, and the backend’s bridged reply passes through conntrack because br_netfilter (net.bridge.bridge-nf-call-iptables=1) hands bridged IPv4 to the ip hooks, which restores the VIP as its source. The applier checks that sysctl and reports a node without it as not converged; every runtime’s node setup must turn it on (docker does).

A node’s own processes (SHC-1378). shcd, an operator’s curl on the node and a network_mode: host container send from the node’s namespace, which never crosses prerouting. chain output hooks nat output with the same dispatch map, so such a flow is dispatched exactly as a container’s on that node: same tiers, same counters. One thing differs. The kernel routes a local packet before nat output rewrites it, so the flow already wears the source of the route to the VIP, which is the uplink’s address. A replica on another node would see that address arrive on shc0, where the peer’s AllowedIPs refuse it, and a replica on this node would see the uplink instead of its bridge. So postrouting masquerades a DNATted flow whose original destination is in the VIP range and whose source is one of the node’s own addresses (fib saddr type local). The flow then wears the outgoing device’s address: the bridge’s for a local pin, shc0’s for a pin elsewhere. Those are the sources a node process gets when it dials the pin itself.

A backend is in a node’s DNAT set only while ready is true, so a replica that is starting, unhealthy, not yet observed, or on a node the cluster no longer sees healthy receives no new connection through the VIP (AC4). Readiness changes already bump the clusterfs revision, coalesced to at most one a second; the vip section’s digest changes only when its content does, and only then does the node re-apply.

This assumes one container per pin at a time, which holds while a replica has one incarnation. The member-specific report that replica-entry pins wait for (section 3) is per incarnation, with draining as a state, so a start-first update that runs two incarnations side by side reads the right one.

Drain before stop (SHC-1155) takes an incarnation out of every routing table (Traefik pools, resolver answers, VIP maps) and waits for the routing nodes to confirm before it stops the copy. The VIP side of that: a draining incarnation is not ready, and after every converge, including one that found the kernel already right, the applier records the clusterfs revision whose vip section its kernel now holds. A node whose table did not change still advances, so it never holds a drain back. How that per-node revision reaches the rollout is SHC-1155’s to choose; the applier exposes it.

As built (SHC-1295), every caller that stops or replaces a replica goes through one primitive, vip.Drainer. A drain ends early only by its own holder’s Release (the verb knows what it drained runs again) or when every live pin it named is released, checked once; otherwise it holds one lease after the verb returns, beaten by its holder, then the holder removes it and bumps; a dead holder’s row is ended by the sweep.

What that guarantees: what a verb stops is out of every routing node’s table before anything is stopped, and stays out for one lease (60 s) after the verb returns. After that, routing follows the nodes’ records. The limit is a node whose recorder is more than a lease behind: its record still says running when the hold ends and the stopped pin is routed to until the record catches up. The fix for that is a desired state the table reads (SHC-1349), not a longer hold.

  • Mark writes one routing_drain row (deployment key, narrowed by service, replica entry or node when set) under an id it mints, and advances the clusterfs revision. The table build carries every backend a row names as not ready, whatever its replica reports, for as long as the row exists. The Drain that Mark returns is the row’s one holder for its whole life: no caller names a row, no drain replaces or ends another’s, so two overlapping stops never undo each other and no deployment status is borrowed as the drain mark. A drain that has to survive a daemon restart is the rollout engine’s own later change (a holder generation on the row).
  • Wait asks every routing node (RPC vip.drained), in parallel, re-reading the routing nodes each round, until each says its kernel holds, converged, a table at the drain’s revision or later with none of those backends ready. A table older than the drain never confirms it, and a node with no applier is not drained: its kernel keeps the last table. After 30 s (placement.DefaultDrainTimeout) the stop goes ahead, X200240VIPDRT names the nodes that had not confirmed, and shc doctor counts it (vip.drain). Begin is Mark then Wait; the rollout engine, which steps, calls Mark and then Pending each step.
  • A drain whose routing nodes cannot be read or whose row cannot be written did not start: no node took anything out of its table. Its Err says so (X503247VIPDRS) and the verb refuses its stop and stops nothing. A row that is written but whose revision could not be advanced did start: no node can confirm a row no revision carries, so Wait tries the bump again each round and waits for it like any other, up to the same 30 s.
  • The row is a lease. Its holder beats it every 20 s. The leader’s VIP sweep removes a row whose beat it has watched stand still for a minute, by its own clock, and advances the revision so every node drops it together; it says so (X200244VIPDRX) and shc doctor counts it. A holder that died is back in routing a lease and one sweep after its last beat under a stable leader, plus however long the leader’s reads and its delete take. The sweeper stamps a beat it first sees with the end of its read and measures up to the start of the next one, so a slow store stretches a lease and never shortens one; a new leader has watched nothing and gives every row a whole lease, so a lease runs out only under a leader that leads for longer than a minute.
  • Release removes the row at once, for what the caller knows is running: a restart the engine answered 0 to and whose target set, read back, is not empty.
  • Hold is for every other outcome: a stop that went through, any verb that failed (a failure does not show that nothing stopped), a restart that matched no container or whose target set could not be read back, a teardown, a verb refused because its drain lapsed. A verb calls it as its last act. When every live pin the drain named is released, read once from the pin ledger, the row goes now: an uninstall, a move’s leg. Otherwise the row holds for one lease from Hold’s return (after the ledger read, which may be slow), still beaten, and then its holder removes it. Nothing a node observed ends a drain: a container record, a replica report and the deployment’s status all follow observations (a recorded stopped is lifted by one live container, SHC-321), and an observation made before a stop can be written after it. A pin is released only by the operation that owns it (a teardown, a move, a job that ended), never by a recorder. The cost: a stop and then a start inside a minute stays out of routing until the hold ends, for a human’s shc stop then shc start (SHC-1340) and for a stop-quiesced backup shorter than a minute (SHC-1350).
  • Nothing but its holder ends a drain whose operation still runs: there is no cap on a drain’s life, which would put what the operation is stopping back in routing under it. shc doctor (vip.drain) names a drain held longer than 15 minutes on the daemon that holds it. The verbs end their drain on every return path, a panic included.
  • A drain whose row lapses under it (the sweep removed it, or its daemon could not reach the store for a lease) says so: Lapsed. Wait stops waiting. A verb that has not stopped anything yet refuses with X503246VIPDRN and stops nothing; a stop already under way cannot be taken back, so the drain says X200245VIPDRO and writes its row again as soon as the store answers. A row no revision carries yet (the bump failed, at Mark or after the row was written again) is bumped again on every beat until one goes through.

Callers today: shc stop and shc restart (the stack or one service), shc uninstall (the teardown, and the stop its safety backup may make first), a backup’s pause-to-stop escalation, and a move’s leg teardown, which drains its own node only because the key is shared with the copy on the other node. A leg that names no node runs on this one and drains this node; a node that cannot say which one it is refuses the teardown (X503248VIPDRU).

Policy holds through the DNAT (AC7). nat prerouting runs at priority -100, before the forward hook where the tenant policy chains sit (SHC-1202, priority -10). By the time the policy sees a VIP packet its destination is the replica’s pin and the real port. A VIP is therefore exactly as reachable as the pins behind it: same tenant on declared ports, own deployment on every port, another tenant never, except the system tier’s accept classes. The bridge anti-spoof guard judges sources, which a DNAT does not touch. The node-plane table’s drop of container traffic to the control-plane range only meets VIP packets that were not DNATted, which refuse_forward has already answered.

The same holds for a node’s own process. Its flow to a VIP is rewritten at nat output and leaves with the source a flow to the pin would have, so the policy judges the two alike. A pin on the process’s own node is reached without crossing the forward hook, and no policy rule sees it. A pin on another node sits behind that node’s fabric_ingress, which admits no node address today: the flow is dropped there by the default-drop, through the VIP exactly as through the pin. The kernel lab names that rule for both.

One policy rule does need to know: the ICMP error accept. The policy lets a pin raise an error only about a flow addressed to it, matched as ct original ip daddr <pin>. On the client’s node the flow’s original destination is the VIP, so an ICMP error the replica raises about a VIP flow (a UDP port unreachable, a fragmentation needed) is dropped there. The accept becomes ct original ip daddr { <pin>, 10.224.0.0/11 } ct reply ip saddr <pin> ip saddr <pin>: the pin answers the flow, and the flow was sent to the pin or to a VIP. A flow without a DNAT matches exactly as before. ct reply ip saddr alone would also admit the pin’s errors about any other DNAT to it, a published port or a hand rule, which the old match refused; the VIP range is the only destination added, and the kernel lab proves that other DNAT stays dropped.

On a fabric that overlaps the VIP range the range is not added (Rule.VIPFlows is false, and the policy’s verify pass re-derives it). VIPs are off there, and the range holds fabric addresses, a node’s own device address among them, so naming it would admit a pin’s error about a published port on such an address. The accept is the pin-only form from before VIPs. A second kernel lab runs that fabric (10.224.0.0/12) and proves the other DNAT stays dropped; it is red with the range always named.

Flows end with their replica (AC4) and with their VIP (AC2). A NAT binding lives in its conntrack entry, so replacing the shc_vip table never breaks an established flow, and also never moves one. After every apply, and once a minute, the VIP applier runs its own pass over the kernel’s table through SHC-1232’s modules/fabric/conntrack. Each DNATted flow whose original destination is in the VIP range names a frontend, its VIP on every port of its protocol, whose targets are the VIP’s running backends; conntrack.End deletes only dumped entries, by reply tuple and kernel ID. Such a flow ends when

  • that VIP is not live in the table (released or unknown), or
  • its backend is not a running backend of that VIP.

A backend that is unready or draining but still running keeps its established flows, like Kubernetes. A stopped one loses them on each node once that node has materialised the revision that says so: with the store reachable, that is the recorder’s observation of the exit, at most a second of revision coalescing, the one-second clusterfs poll, and the apply and pass that follow the changed section. A node that missed a publication recovers on its next successful clusterfs refresh, which materialises, applies and runs the pass; the minute pass only repeats the judgement over what the node already holds, for entries created after the last one. A node cut off from the store has no bound and keeps its last revision. SHC-1232’s own pass already ends a flow whose backend pin was released and one the policy no longer accepts; it judges by the backend, so VIP flows are covered there too.

  • No Go in the data path (AC8). The kernel’s nft table and conntrack are the only forwarding state. shcd computes and installs; it is never on a packet’s path.
  • shcd restart. The table stays in the kernel while shcd is down. On start the applier reads current/vip/services.json from disk, needs nothing from the control plane, compiles and finds the marker already matching: no-op. If the table changed while shcd was down, one atomic replace applies the new one, and established flows keep their bindings through it.
  • Partition. A node cut off from the store keeps serving its last revision, the same as the resolver.
  • nft flush ruleset by hand. The marker goes with the rules; the next tick re-applies.
  • Leader change. Allocation state is rows, not memory. A new leader’s sweep reads the same rows; two racing allocators collapse on the UNIQUE indexes.
  • No fabric in the revision. A node with no clusterfs yet, a revision without the vip section, or a cluster whose posture is not shc has a table that names no fabric. The applier then loads no rules, sets no hairpin or sysctl and runs no flow pass; it only drops a shc_vip table an earlier revision installed. The posture is read from the materialised revision, so this holds offline and across a restart. A container’s packet to the range meanwhile meets the node plane’s drop of the control-plane range.

<service>.<stack>.<environment>.<tenant>.shc.internal, the bare name through dns_search, and a plug’s provider name all answer with the service’s VIP (AC6). The answer does not depend on readiness: the VIP itself refuses fast when no replica is ready, and a cached answer stays right through every reschedule. Replica-level names stay out of scope until groups and members are ruled. shcd answers DNS (SHC-1213); that lane owns the change, reading the VIP from the store or the vip section.

shc cluster fabric vips [-s stack] [-e environment] [-o json] lists each service’s VIP, its backends with node and readiness, and the tier this node dispatches to, plus the version this node’s kernel holds. It reads the same section the applier compiles, so what it prints is what the kernel does (AC1).

ce:

  • modules/vip/compile: pure. The file types, range math, tier selection and the nft renderer. No store, no exec.
  • modules/vip: the service_vip store and allocator, the leader sweep, the clusterfs section, and the node applier (nft converge, bridge port hairpin, the flow pass, the recorded revision). Readiness is injected, so the package does not import modules/app.
  • ent/schema/service_vip.go and the squashed migration.
  • subsystems/impl/service_vip.go: two subsystems, service_vip_alloc (the allocation sweep, leader only) and service_vip (the node applier, every node). The section is added to the clusterfs subsystem’s list.
  • modules/vip.LiveVIPs: the live VIPs through a clusterfs build’s reader, for the resolver section (SHC-1213). It answers nothing while VIPs are off, so a name never resolves to an address no node routes.
  • modules/cluster: the vips route and command.

shc: tests under tests/_ce/modules/vip, the subsystem catalog count, the pin bump, and the kernel proof.

Unit tests, table-driven, red first: the allocator (cursor order, uniqueness, release, tombstones, wrap with the top slot still held, the 24 h floor, range full, concurrent allocators), the sweep (allocate missing, release gone, a failed read releases nothing), the section (backend mapping for web and web-r<N> pins, positive readiness, running, stable ordering), the flow pass predicate, and the compiler (each tier, no ready backend, hairpin elements, address-only DNAT, name and comment validation, version stability).

A kernel lab in Go, gated like the fabric’s existing ones (root, Linux): three network namespaces stand in for three nodes in two locations, each with its own bridge, nftables and conntrack, routed over veth pairs. It loads the compiled VIP documents together with the policy the SHC-1202 compiler renders for the same graph, and proves AC3 (cross-node), AC4 (unready gets nothing new, a stopped backend’s flow ends), AC5 (counters on the local and location tiers), AC7 (another tenant is dropped after the DNAT), the hairpin, a same-bridge pair and AC8 (a long-lived connection survives a table replace). It also proves the ICMP cases: a replica’s error about a VIP flow reaches the client, a replica’s error about its own flow through its VIP comes back to it, an error about any other DNAT to the pin is dropped, and another tenant cannot forge one. A counter put on the responder accept in the lab says that rule took each error, and with the accept removed the cross-node one is dropped. (The hairpin’s error is a packet from the replica to itself once the masquerade is undone, which its deployment’s own accept would pass too.)

A multi-node proof script, POSIX sh, for any Debian 13 hosts reachable over ssh: it builds the documents with the same compiler, sets up WireGuard, bridges and namespace containers on each, and runs the same checks with counters and transcripts, the hairpin and the three ICMP cases included. The live proof on a real cluster is owed by the rig lane (SHC-1123).

  • SHC-1213 (shcd DNS): answer service names and provider names with the VIP (section 8).
  • SHC-1202 (tenant policy, merged): the ICMP error accept binds the responder as ct reply ip saddr and admits a VIP as the flow’s original destination beside the pin itself (section 6). Nothing else.
  • SHC-1232 (pins, conntrack): nothing required; the VIP’s flow pass uses its conntrack package.
  • SHC-1155 (drain before stop): read the applier’s recorded revision as the VIP’s routing confirmation (section 5).
  • SHC-1144 / SHC-1216 (replica projects): keep filing each replica entry’s pin as <service>-r<N> under the deployment key; the section maps those to the service. Replica-entry backends stay unready until nodes report each pod member’s own state per incarnation, with a clusterfs bump on every accepted change (section 3).

Nothing here needs a ruling to build. Two follow-ups are worth naming: whether a UDP flow to a backend that went unready but still runs should end (Kubernetes ends it; here it keeps going until the backend stops), and whether shc cluster fabric explain should accept a VIP as an end and judge it against the backend the asking node would pick.