Hookshot's first deployment: what a live estate settled
Status: open design. A second appservice registration, hand-run job context, and the transform render arms are still owed.
Companion to 2026-08-09-matrix-notification-bus.md (the design) and
2026-08-10-hookshot-phase0-readiness.md (the ruling map). Those two
documents end with the same sentence in different words: the package has
never been deployed. It has now. This document records what the deployment
measured, what it changed in the package, and what each producer’s edge
proved on the far side, bitspur corp/prod, 2026-08-14/15.
The ruling this lane executes is the notification-bus row of
docs/decisions/2026-08-14-capability-and-backlog-rulings.md: “Hookshot
for everything: one delivery path: wazuh, gitlab and GlitchTip all route
through the hookshot bridge”, which overrode §9.1’s phase-0 recommendation
(native Matrix for gitlab, a custom- script straight to the client-server
API for wazuh). Every producer below therefore rides the bridge, including
the two the proposal argued should not.
1. ★ The finding that reshaped the package: url was already taken
Section titled “1. ★ The finding that reshaped the package: url was already taken”matrixAppservice is ONE registration shared by every consumer, and the
registration’s url (the address Synapse pushes transactions to) is a
single string authored on the provider. Measured on bitspur before anything
was installed:
$ docker exec corp_prod_matrix-synapse-1 cat /data/appservices/shc.yaml{ "id": "shc", "url": "http://matrix:8090", "sender_localpart": "_shc_appservice", … }$ docker inspect … | grep 10.0.44.126/corp_prod_huly-matrix-1 aliases=[… matrix prod.matrix.huly]http://matrix:8090 is huly’s two-way Matrix bridge, deployed and
working (43h uptime, its log shows inbound-event handling). The readiness
doc’s §2.4 called two consumers on one registration a standing risk; on a
live estate it is not a risk but an allocation; the pointer is spent, and
hookshot cannot have it without taking it from a running product surface.
So hookshot is deployed SEND-ONLY, and that is sufficient, because of
what the bridge actually needs an inbound transaction for. Measured in the
pinned image (ghcr.io/matrix-org/matrix-hookshot:7.4.4):
| where a static connection is built | trigger | needs a transaction? |
|---|---|---|
BotUsersManager.getJoinedRooms() → Bridge.js:459 → createConnectionsForRoomId(roomId, false) | startup, over the rooms the bot is already in | no, a client-server GET /joined_rooms |
Bridge.js:793 onRoomJoin → createConnectionsForRoomId(roomId, true) | a pushed membership event | yes |
Posting is likewise a client-server call with the appservice token. The only thing the transaction half buys a generic-webhook bus is the auto-accept of an invite, and an appservice can simply join itself.
1.1 What changed in the package
Section titled “1.1 What changed in the package”The invite + join moved into the boot script, before exec, and
hookshot-init.job.yaml was deleted rather than left beside it (M12: the
replaced surface goes in the same program). The ordering is the whole point:
- a
post_installhook job necessarily runs after the bridge has finished enumerating its rooms, so an invite issued there converges one restart too late, measured exactly that way on the first install, where the job’s invite landed at 23:37:39 and the bridge kept reportingFound 0 connections; - the boot script invites as the alerts bot (
matrixsocket), joins as the appservice with its own token, and reads backjoined_membersbefore starting the daemon.
First boot after the change:
hookshot boot: rendered config with 3 static connection(s)hookshot boot: hook glitchtip -> http://prod.hookshot.hookshot:9000/webhook/844abdf5... (rest masked …)hookshot boot: invited @_shc_appservice:matrix.corp.bitspur.com to !IqVzZGjkiYuEdXYyiA:…hookshot boot: read-back confirms @_shc_appservice:matrix.corp.bitspur.com JOINED !IqVzZGjkiYuEdXYyiA:…INFO [Bridge] Bridge is now ready. Found 3 connectionsand the acceptance ladder’s step 4 (§9.2) answered on the first try:
POST /webhook/844abdf5... -> 202 {"ok":true}… @_shc_appservice:matrix.corp.bitspur.com m.room.message "shc lane probe: hookshot generic webhook, glitchtip hook"★ This also closes readiness §4 unknown 7: a URL derived by the recipe has
never been POSTed to a bridge whose config rendered the same recipe. It has
now, and the glitchtip edge derived the identical id in Python
(844abdf5-a19f-4123-b425-d990f221b9e9 on both sides).
2. The other measurements the first deployment produced
Section titled “2. The other measurements the first deployment produced”| # | measured | consequence |
|---|---|---|
| D1 | --var hooks=[{"name":…}] cannot set a list-typed var: --var values arrive as strings, and jinja iterated the JSON text character by character, rendering hooks named '' and crash-looping the container | vars.hooks is now a STRING spec, name=room;name=room, which is the wire format compose always flattened to anyway |
| D2 | a MAS-fronted Synapse refuses an appservice registration without inhibit_login: 400 IO.ELEMENT.MSC4190.M_APPSERVICE_LOGIN_UNSUPPORTED | the boot script’s register call sets inhibit_login: true; the bridge wants no device and no access token, it has the as_token |
| D3 | the appservice bot joining the alerts room makes that room appservice-interesting, so huly’s bridge now also receives its transactions | observed: no errors, no behaviour change in corp_prod_huly-matrix-1 over the following half hour. Recorded because it is the shared registration showing through |
| D4 | shc job run <hook job> renders the job with neither the deployment’s vars nor its integration context; the same job that had MATRIX_TOKEN, ROOM_ID and HOOKS_SPEC during install came back empty-handed when run by hand | a platform defect worth its own lane: a hand-run job that silently sees defaults is a debugging trap. It is not a blocker here; the package no longer has a hook job |
| D5 | wazuh-control start inherits the job’s stdout through the daemons it forks, so a job that lets it write to the pipe never sees EOF and burns its timeout | the wazuh edge redirects it to a file; start (not restart) leaves the running SIEM alone and brings up only integratord |
3. Per-producer verdicts, each with its far-side proof
Section titled “3. Per-producer verdicts, each with its far-side proof”All three ride the bridge, per the ruling. Each edge is a webhook.plug.yaml
with a post_deployment job that converges the producer’s OWN notification
surface and an unintegration job that removes it.
3.1 GlitchTip: bound, converged, proven
Section titled “3.1 GlitchTip: bound, converged, proven”apps/flagship/glitchtip/webhook.plug.yaml. The job writes an
AlertRecipient of type webhook through the ORM (manage.py shell),
because the package seeds no API token and the REST surface has no standalone
recipient endpoint (readiness M3). One dedicated ProjectAlert per project,
named shc-matrix, so the edge never bolts a row onto an alert a human made.
SSRF posture: ruled in favour of the in-fabric URL. GlitchTip refuses a
private-IP recipient twice (pydantic validator at creation, check_url_safe
at send) unless GLITCHTIP_ALLOW_PRIVATE_IPS is set; default False. The
alternative was a public vars.urlBase on the hookshot stack, i.e. a bearer
secret on the internet. The package gained allowPrivateWebhookIps (default
"false"), the job refuses loudly when the URL is private and the flag is
off, and the estate is bound with it true.
Far side, from a real ingested error event through GlitchTip’s own alert path:
2026-08-15T00:03:12Z @_shc_appservice:… m.room.message "GlitchTip Alert"(The issue detail rides attachments, which hookshot carries in
uk.half-shot.hookshot.webhook_data rather than the visible body, the
measured text-only render, with JS transformation functions off by policy.)
3.2 Wazuh: bound, converged, proven
Section titled “3.2 Wazuh: bound, converged, proven”apps/flagship/wazuh/webhook.plug.yaml, and it is the one the proposal
warned is “larger than it looks”. The job writes
/var/ossec/integrations/custom-shc-webhook + .py (750 root:wazuh, the
shipped convention) into the integrations volume and inserts a marker-fenced
<integration> block into ossec.conf; there is no env var and no drop-in
directory, and wazuh-integratord does not exist as a process until such a
block does. Volume-written rather than image-baked because this package does
not build the vendor image; re-asserted every deploy so an image upgrade that
reseeds the volume converges instead of silently dropping the route.
vars.webhookAlertLevel (default 10) is the only volume control the edge
has; the bridge renders every POST it receives.
Far side, from nine synthetic failed-SSH lines (rule 5712, level 10):
2026-08-15T00:15:22Z @_shc_appservice:… m.room.message "[wazuh] level 10 sshd: brute force trying to get access to the system. Non existent user. on 489b7a0c87f2 (rule 5712) - /var/ossec/logs/active-responses.log"★ One diagnostic worth keeping: before the edge was bound, the same script
failed with POST failed: <urlopen error [Errno -2] Name or service not known>; the manager could not resolve prod.hookshot.hookshot until the
plug put it on the bridge’s network. The bind is what makes the name exist,
which is the argument for proving edges by binding them rather than by
rehearsing their commands.
3.3 GitLab: edge built, bind REFUSED by an orchestrator boundary
Section titled “3.3 GitLab: edge built, bind REFUSED by an orchestrator boundary”apps/flagship/gitlab/webhook.plug.yaml drives GitLab’s instance-level
Slack-notifications integration through gitlab-rails runner, which emits
the Slack incoming-webhook dialect (text + attachments), the dialect the
socket advertises and hookshot renders natively. Instance level, not per
project, because a bus that carries only the projects somebody remembered to
wire is not a bus; the job propagates the instance row so projects inherit
it. Events: issues, merge requests, tag pushes and pipelines
(webhookBrokenPipelinesOnly, default true). Push, note and wiki events stay
off; they are the firehose.
★ The file states plainly that this edge overrides §6.3’s recommendation (GitLab 17.3+ has a native Matrix integration) on the owner’s ruling: one delivery path for the whole estate beats three that fail differently.
★★ AND THE BIND DOES NOT LAND ON THIS ESTATE, for a reason that has
nothing to do with either package. Measured, shc update gitlab -i webhook=hookshot:
gitlab: degradedshc: [X500059DCKCMP] docker stack failed, exit status 1, output …network "corp_prod_hookshot_default" is declared as external,but it is not in the right scope: "local" instead of "swarm"The update rolled back cleanly (gitlab running, its other three edges
intact). gitlab is a SWARM service (node-ca97810c782b, deployed with
docker stack deploy); hookshot is a compose deployment on
node-d5879f311b04, so the network an integration attaches a consumer to is a
LOCAL bridge. A swarm service cannot attach a local-scope network, and no
amount of plug authoring changes that; this is a placement fact, and the
first time the estate has been asked to wire a swarm consumer to a compose
provider.
The fork, stated rather than taken, because each arm costs something an edge may not spend on its own:
- Place hookshot on the swarm. Its network becomes an attachable
overlay, which BOTH orchestrators can join; compose containers can
attach an attachable overlay, which is why this is the only arm that
serves all three producers at once. Costs a
shc moveof a stack whose other two edges are proven and would be re-rendered. - Publish the listener on the node and advertise that as
urlBase. Reachable from every node over the private network, at the price of a bearer URL that is no longer confined to the fabric, andurlBaseis provider-global, so it changes the posture for glitchtip and wazuh too. - Public ingress
urlBase: the §6.1 posture this estate’s SSRF ruling already declined; it puts the bearer secret on the internet.
Until one is chosen, GitLab’s own notifications stay unrouted; nothing was left half-configured on the GitLab side (the failed update never reached the job, and the instance-level integration row does not exist).
3.4 Same day, second pass: the boundary resolved WITHOUT the move
Section titled “3.4 Same day, second pass: the boundary resolved WITHOUT the move”Arm 1’s intent (hookshot’s network becomes an attachable overlay both
orchestrators can join) turned out not to need shc move at all, because
the literal move has a landmine the fork missed: a swarm-runtime stack’s
default network is an overlay but not attachable, so moving hookshot
onto the swarm would have handed gitlab its network while cutting off the
two standalone-compose producers. The engine already owns the correct
mechanism: ensureSwarmAttachableNetworks materialises any AUTHORED
non-default network as an --attachable overlay when swarm is active (it
deliberately never touches <project>_default), and this estate has proven
that arm live for months (gitlab, a swarm service, reaches its
compose-deployed postgres over corp_prod_gitlab_db_public).
So the hookshot service now authors public beside default
(apps/standalone/hookshot/compose.yaml). One shc update hookshot later,
corp_prod_hookshot_public existed as swarm overlay attachable=true, the
socket-hostname alias (prod.hookshot.hookshot) landed on it, the bridge
came back with its 3 connections, and shc update gitlab -i webhook=hookshot
sailed through the very docker stack deploy that had refused; the gitlab
task carries the overlay as its fifth network. glitchtip and wazuh were
re-rendered onto the overlay and both re-proven end to end (wazuh
2026-08-15T01:41:30Z, glitchtip 2026-08-15T01:57:15Z, same room). The SSRF
posture is untouched: urlBase stays in-fabric.
The bind then measured two more GitLab-18.x realities the plug had not
met, both surfaced by one save! when the job finally ran:
Validation failed: Organization can't be blank, Exactly one of project_id,group_id, organization_id must be present, Webhook is blocked: Requests tothe local network are not allowed- instance-level integrations are organization-owned in 18.x; the job
now assigns
Organizations::Organization::DEFAULT_ORGANIZATION_IDwhen the row carries none; - GitLab has its own SSRF gate, the exact counterpart of glitchtip’s
GLITCHTIP_ALLOW_PRIVATE_IPS: a local-network webhook is refused at save unlessallow_local_requests_from_web_hooks_and_servicesis on. The package gainedvars.webhookAllowLocalNetwork(default"false"): true lets the job enable the setting loudly, false makes it refuse loudly, the same consent shape the glitchtip edge ruled, and the estate binds with it true.
Two operational traps this pass measured, worth their own records:
- D6: a bare
shc updateon a version-unpinned deployment re-renders it from the CATALOGUE (X200971UPDHED warns; X200093APPORP names the dropped files). The three producer stacks ran--from <local path>packages, and one bare update silently reverted glitchtip’s app tree; theGLITCHTIP_ALLOW_PRIVATE_IPSenv vanished and every alert died at the send gate (“webhook delivery blocked: URL resolves to a private/internal IP”) while the recipient row still looked configured. Until a release carries these packages, every update of the three producers must restate--from. - D7: a consumer whose only render change is a provider network’s
external NAME is not recreated by
docker compose up: the service’s own config hash keys on the network KEY (__shc_int_2), not the external name under it, so glitchtip kept running on the dead local bridge while its rendered compose said overlay. One--force-recreateconverged it; wazuh, whose render changed in other ways too, recreated on its own.
3.5 Same day, third pass: GitLab producer proven; the silent-skip landmine
Section titled “3.5 Same day, third pass: GitLab producer proven; the silent-skip landmine”The §3.4 bind left the instance Slack row active (org=1,
allow_local=true, in-fabric URL) and (despite
Admin::PropagateIntegrationService being nil on 18.11) all six
projects inherited it (inherit_from_id set, same masked URL). 18.x
save! of an instance integration enqueues propagation on its own, so
the job’s if defined?(Admin::…) skip was a no-op that happened to
converge. That is not a contract: a GitLab that does not auto-enqueue
would report success with zero inherited rows, which is the silent
all-alerts-dropped edge the second pass named. The job now calls
Integrations::PropagateService.new(integration).execute (the living
surface, defined? == "constant" on this estate) and raises if
neither constant exists.
Far-side producer proof, 2026-08-15T03:03:13Z, from swarm02 (gitlab’s
node) through the overlay: Integrations::Slack#execute of an issue
payload returned true; hookshot logged onGenericHook for the gitlab
hook id and MatrixSender Sent event; the room gained:
2026-08-15T03:03:13Z @_shc_appservice:… m.room.message "Received webhook data: ```json { \"payload\": \"{\\\"fallback\\\":\\\"[<https://git.corp.bitspur.com/root/clone-proof|root/clone-proof>] Issue …"Reachability was independently confirmed from the gitlab container:
prod.hookshot.hookshot → 10.0.93.2, POST /webhook/shc-health-probe
→ 404 {"ok":false,"error":"Webhook not found"} (listener up, unknown
id refused). Sidekiq was running.
★ Render gap, measured: GitLab’s Slack integration POSTs the incoming-webhook
dialect as application/x-www-form-urlencoded with a payload=<json>
field, not a JSON body with top-level text. Hookshot’s generic handler
therefore takes the “Received webhook data” dump rather than the text
extract. Delivery is the ruling’s bar and it cleared; making the visible
body a one-line issue title is a transform/Content-Type follow-up, not
a missing producer.
3.6 Same day, fifth pass: the render gap closes producer-side
Section titled “3.6 Same day, fifth pass: the render gap closes producer-side”The follow-up §3.5 called for was assumed to live on the bridge. It did
not. GitLab’s chat notifiers all build the same
Integrations::ChatMessage::* object and differ only in how they encode
it, and Integrations::Pumble#notify (18.11 omnibus,
app/models/concerns/integrations/base/pumble.rb) posts
header = { 'Content-Type' => 'application/json' }Clients::HTTP.post(webhook, headers: header, body: Gitlab::Json.dump({ text: message.summary }))Exactly the one shape transformHookData renders. Both rows were live
against hook 9ddb6e23-… briefly, which measured the difference with no
other variable: issue #5 at 2026-08-15T07:51:56Z produced two room
messages 15 ms apart, …889Z the sentence (Pumble) and …904Z the fenced
dump (Slack).
The edge moved to Pumble, and the converge job destroys any
instance-level Slack row whose webhook is on this bus, scoped by URL
prefix so an operator’s real Slack row is never touched, and destroyed
rather than deactivated because two active notifiers on one hook id
double-deliver every event (M12: the replaced surface goes in the same
program). Run verbatim from the plug against the live instance at
07:58:40Z: removed 1 superseded instance-level Slack row(s) pointing at this bus, seven Slack rows (instance + six inherited) gone, seven Pumble
rows in their place. Re-running it is a no-op.
Fresh proof at 07:59:32Z: a real issue (root/clone-proof #6) produced
exactly one onGenericHook (08:00:12:355), one MatrixSender Sent event
(08:00:12:428), and one room message:
2026-08-15T08:00:12.394Z @_shc_appservice:… m.room.message "[<https://git.corp.bitspur.com/root/clone-proof|Administrator / clone-proof>] Issue <https://git.corp.bitspur.com/root/clone-proof/-/work_items/6|#6 hs-r5b-075931 far-side proof> opened by Administrator (root)"gitlab: AMBER → GREEN, with no transformation function. The
allowJsTransformationFunctions ruling is still owed, now for glitchtip
alone; all eight GlitchTip 6.2.5 RecipientTypes were re-read at the
running pin and none carries issue detail in a top-level string text or
html.
3.7 The edge ran from the package, not from a transcript, and what that exposed
Section titled “3.7 The edge ran from the package, not from a transcript, and what that exposed”§3.6’s converge was run verbatim out of the plug file against the live
instance. That proves the Ruby and not the packaging, and the difference
was measurable: the gitlab deployment’s stored app tree
(/var/lib/shc/deployments/corp/gitlab/prod/app/) was still a revision
that predated §3.5, so it wrote Integrations::Slack; every future
replay of post_deployment, for any reason, would have put the
form-encoded row back beside the Pumble one and started double-delivering.
A converged estate was one unrelated shc update away from the defect it
had just fixed.
So the landed package was pushed to the estate’s local app dir and the
real update was run: shc update gitlab -t corp -e prod --from /home/admin/gitlab-app -i webhook=hookshot, 2026-08-15T19:24:50Z, job
configure-matrix-webhook on service gitlab at 19:25:46Z, stack back to
running, exit 0. Stored plug b5875286 → b5ecd5ec (the landed bytes),
app_digest 3fe052f2 → 0f2f408b. The row state is what proves which
job ran: afterwards the instance row is still the single
Integrations::Pumble on the bus URL with six inherited children and
zero Slack rows; the tree it replaced would have created one.
Fresh far-side proof, all three producers fired after that update, each
landing exactly one onGenericHook and one MatrixSender Sent event:
19:28:22.985Z [wazuh] level 10 sshd: brute force trying to get access to the system. Non existent user. on 70f94003ca68 (rule 5712) - …/active-responses.log19:29:03.797Z GlitchTip Alert19:29:14.996Z [<…|Administrator / clone-proof>] Issue <…/work_items/9|#9 r7-192819 far-side proof> opened by Administrator (root)wazuh GREEN, gitlab GREEN (one message, no dump twin), glitchtip AMBER: delivers, renders the constant banner, and that is the one edge the transformation-function ruling is still owed for.
★ AND THE PUBLISHED CATALOGUE IS THE LANDMINE D6 DESCRIBED, ARMED.
Measured 19:20Z: the estate’s shc repo
(https://selfhostedcloud.org/apps/index.yaml, synced 18:02:18Z) carries
the Slack-writing gitlab edge
(Integrations::Slack.where(instance: true).first_or_initialize at line 138
of its webhook.plug.yaml), and shc update gitlab --check reports the
deployment STALE against it. Until a release publishes the fix, a bare
shc update gitlab on any estate re-renders the form-encoded edge, which
renders every alert as a dump and, beside a live Pumble row,
double-delivers. Restate --from on every update of this stack until a
release carries the package.
4. What the guards now hold
Section titled “4. What the guards now hold”apps/standalone/hookshot/tests/tests.bats: 26 tests, including four new ones: no hook job may reintroduce the too-late invite; the boot script invites, joins with the appservice token, and reads back; the convergence is never fatal and never blocks the boot; derived URLs are printed masked. Two guards that had been silently passing over a nonexistent path since the apps/ re-tiering ('..', 'matrix'; the provider isapps/flagship/matrix, not a sibling) now resolve the provider by name and fail if it moves again.tests/unit/catalog/webhook_derivation_parity_test.go: new. The derivation is now written in three languages (JavaScript on the bridge, Python on the glitchtip and wazuh edges, Ruby on the gitlab edge), and a single divergent nibble produces a URL that 404s forever while every stack reports itself configured. The test pins the canonical VALUE (recomputed in Go from the recipe’s definition) and the canonical TEXT of each language’s spelling, and additionally requires everywebhook.plug.yamlto declare a[a-z0-9-]+hook, to carry an unintegration job, and to print its URL masked.- (second pass) the hookshot bats guard that pinned “default network only”
now demands
defaultANDpublic; droppingpublicreopens exactly the §3.3 scope refusal for every swarm-runtime producer while the compose producers keep working, which is the asymmetry that earns it a name. Andapps/flagship/gitlab/tests/tests.batsgained a guard pinning theorganization_idassignment, the consent-var wiring, the SSRF-gate name, the loud refusal, and the livingIntegrations::PropagateServicecall ending in a raise (not adefined?skip); mutation-proofed red-by-name.
5. State at hand-off, and what is still owed
Section titled “5. State at hand-off, and what is still owed”On the estate (bitspur corp/prod), after the §3.5 third pass: hookshot
running with three static connections (glitchtip, wazuh, gitlab) into the
alerts room !IqVzZGjkiYuEdXYyiA:matrix.corp.bitspur.com, its public
network an attachable swarm overlay every producer attaches; all three
producers bound and proven end to end over that overlay: glitchtip
00:03:12Z and 01:57:15Z, wazuh 00:15:22Z and 01:41:30Z, gitlab 03:03:13Z.
The §3.3 boundary is closed by the authored provider network, not by a
move.
The one drift this lane keeps deliberately: the producer stacks were
brought up from --from <local path> packages so the estate could run the
edges before the catalogue carried them. D6 above is what happens when
anyone forgets: a bare shc update reverts the tree to the catalogue and
the edge dies in whichever way its package machinery was holding open.
Once a release publishes these packages, each stack should be updated back
onto the catalogue app; until then, restate --from on every update.
★ Corrected 19:20Z (§3.7): “the stacks run --from packages” is not a
property the estate records, and only hookshot has it. Read off the
manifests: hookshot carries app_name=/home/admin/hookshot-app, while
gitlab, glitchtip and wazuh all carry catalogue app names; their
local-path origin left a stale snapshot, not a pin. --check says so:
gitlab and glitchtip are STALE against the published catalogue and
wazuh is CURRENT with it. So the mitigation is a habit, not a
mechanism, and the catalogue it would fall back to still carries the
form-encoded gitlab edge (§3.7).
Still owed:
- A second appservice registration on the matrix package (§11.4, now
concrete rather than hypothetical). Until it exists, whichever consumer
binds first owns
urland every other consumer is send-only. That is survivable for a webhook bus and fatal for anything that needs inbound events. shc job runcontext (D4): a hand-run job that silently renders defaults.- The
htmland fenced-JSON render arms oftransformHookData, andenableHttpGet: false→ 405, remain source-read only (readiness §4.4/4.5). GitLab’s form-encodedpayload=Slack body is now a measured sibling of that gap (§3.5): the event lands, the visible body is the generic dump. - Nothing here touched promanager; the same two-stack bring-up applies there
and its matrix stack’s
appservice.urlshould be read before assuming it is free.