Skip to content
SHC Docs

Retire the Frappe/Python-era API wire dialect

Status: open design. Step 1 shipped. The daemon and CLI flag-day, steps 2 and 3, are not done.

Proposed2026-07-14 (sweep-legacy decision)
Step 1: provider dual-dialect shimSHIPPED 2026-07-14, shc 880b07a3b
Step 2: daemon + CLI flag-dayowed
Step 3: drop the shimowed, optional
Re-measured againstce d054d79, shc 184dc7c99 (2026-08-07)

This document was written on 2026-07-14 and its step 1 landed the same day, but the document itself was never landed; it sat on an unmerged branch while eight comments in shipped code cited it by path (pkg/apiclient/client.go and pkg/apiclient/envelope_internal_test.go in both repos). This revision lands it and re-measures every count against current main, because the CE/EE split moved all of the code it describes and changed the sequencing conclusion.

The daemon still speaks the two success envelopes the Python port inherited from Frappe: {"message": …} on the /api/method/* surface (WriteMessage) and {"data": …} on the /api/resource/* single-doc surface (WriteDoc), both in ce core/api/respond.go. Errors ride a third Frappe shape: {exc_type, code, message, http_status, _server_messages, exc} (coded.Envelope, ce core/coded/transport.go, written by ErrorHandler/WriteError). Every consumer is in-house: the shc CLI (plus the x9s TUI) and the terraform provider. Under the no-backcompat doctrine this is a clean kill-candidate, nothing external parses these bodies, but it touches the daemon response layer AND every client decoder at once, and one of those decoders is version-decoupled from the daemon. So it needs a migration plan, not a blind kill.

The target already exists in-tree: the newer REST-style routes (/api/dns, /api/user, /api/health, …) skip the envelope entirely (WriteJSON; envNone in pkg/apiclient/client.go) and are the de-facto dialect this proposal promotes to universal.

Where the code lives now (post-CE/EE split)

Section titled “Where the code lives now (post-CE/EE split)”

The 2026-07-14 draft used the pre-split layout (internal/core/api/…, one module, one repo). That layout is gone. Current homes:

ThingRepoPath
WriteMessage / WriteDoc / WriteJSON / envelope structscecore/api/respond.go
WriteError / ErrorHandler, coded.Envelopececore/api/error_handler.go, core/coded/transport.go
Generated API client + the shim (CE route surface)cepkg/apiclient/client.go
Generated API client + the shim (full/EE route surface)shcpkg/apiclient/client.go
The client the terraform provider actually importsshcgitlab.com/selfhosted-cloud/shc/pkg/apiclient
CLI / x9s decodersbothper-module daemon_client.go, types.go

internal/ no longer exists in either repo (0 files under it in shc), so every internal/core/... and internal/modules/... path in the original draft should be read with that prefix removed and the repo column above applied.

The surface (re-measured, non-test files, at the SHAs above)

Section titled “The surface (re-measured, non-test files, at the SHAs above)”

Server writers: the wrapped surface now spans BOTH repos, because shc’s EE modules call ce’s api package:

writerceshctotal
WriteMessage → {"message": v}421254
WriteDoc → {"data": v}213
WriteJSON → raw v (the target shape, already live)13114

So 57 wrapped call sites, up from the 42 the draft measured; the surface grew, and a third of the growth is on the shc side, which did not exist as a separate repo when the draft was written.

Client decoders. Files carrying a json:"message" or json:"data" decode tag, non-test: 90 in ce, 25 in shc. The CLI/x9s side still has no choke point: the unwrap is hand-rolled per module in daemon_client.go files and generic Message[T] / Data[T] / ListResponse[T] wrapper structs. The one thing that IS centralized on the CLI side is the error envelope: daemonHTTPErr → coded.FromEnvelope reconstructs the typed error in one place.

Provider decode. The entire success-decode is the env switch in invokeJSON selecting one of envNone | envMessage | envData; the entire error-decode is DecodeError. Generated call counts:

envMessageenvDataenvNonetotal
ce pkg/apiclient1441476234
shc pkg/apiclient (what the provider imports)1821578275

Two functions still cover every provider call, which is what made the shim cheap, and they are now duplicated across the two generated clients, so step 1 had to land in both. It did.

Two warts the map exposes, both worth fixing in the same sweep:

  • data is overloaded. docEnvelope.Data is a pure single-key wrapper, but ListResponse.Data is a field of a list payload that also carries next_cursor/has_more/total_count siblings. De-wrapping is therefore not “unwrap to the inner value” uniformly: a single doc collapses to the bare object, but a list must keep its pagination-bearing struct at top level. The target must be stated as “the response object, un-nested,” not “the value under .data.”
  • Double-wrapping. Several app router sites call WriteMessage(w, map[string]any{"data": data}) → the wire is {"message":{"data":…}}. This is the Frappe muddle at its worst and disappears the moment the message envelope is removed.

Promote the shape that the envNone calls and 14 server routes already speak:

Success = the response object as the top-level JSON body. No message wrapper, no data wrapper. Lists carry their items plus pagination as sibling fields of that object; single docs are the bare object. Failure = a single consistent error object. Recommended: {"error": {"code", "message", "details"}}, replacing the six-key Frappe error envelope (exc_type, exc, _server_messages, http_status all drop; http_status is already redundant with the HTTP status line, and the daemon-side warning path keeps its 204 + X-SHC-Warning-Code header unchanged).

Server-side this means: delete WriteMessage, WriteDoc, messageEnvelope, docEnvelope; every current caller becomes a WriteJSON. WriteJSON (already funnelling encode-failures through the coded handler as X500938RSPENC) becomes the sole success writer.

The error-envelope change is separable from the success-envelope change (its decode is two choke points, not 115 files) and MAY be sequenced as its own phase, or dropped entirely from this pass; see non-goals.

The coupling that decides sequencing (the crux)

Section titled “The coupling that decides sequencing (the crux)”

The client families have different version relationships to the daemon, and this, not code size, dictates the migration. The split added a third one.

  • The CLI ships inside the same .deb as the daemon. shc and shcd are one artifact, built from one commit, installed together. An operator NEVER runs shc vX against shcd vY where X≠Y; the deb makes it impossible. For the CLI boundary, a synchronized flag-day is safe by construction.

  • The terraform provider is a separately versioned artifact. An operator pins a provider version in required_providers and points that pinned plugin at an estate whose daemon runs whatever deb happens to be installed, upgraded on a different cadence. So provider vN genuinely talks to daemon vM, N≠M. This is the ONLY real cross-version wire surface in the system, and it is exactly where a naive flag-day breaks in both directions: estate upgraded first → old provider’s envMessage unwrap finds no .message and every resource read errors; provider bumped first → new decoder chokes on the old daemon’s wrapper.

  • NEW since the draft: ce and shc are two repos on one wire. The writers live in ce; 13 of the 57 wrapped call sites live in shc. A daemon flag-day is therefore a lockstep two-repo change, not a single commit. It lands ce-first, then shc with the ce pin bumped (the landing order this repo already mandates) and the shc pin bump is the only synchronization point where both halves are observable together. An shc that pins a flipped ce while still calling api.WriteMessage simply fails to compile (the symbol is gone), which is the desired guardrail: the break is a compile error, not a wire mismatch.

Verdict: flag-day the CLI (safe, same artifact), land the daemon flip ce-then-shc in one pin bump, and keep the provider boundary dialect-tolerant. That tolerance is now in place.

Step 1: provider shim. DONE (shc 880b07a3b, 2026-07-14; mirrored in ce). invokeJSON’s envMessage/envData arms route through UnwrapEnvelope(raw, key), which peeks the top-level object: if the wrapper key is present it unwraps, otherwise it returns the body as-is. DecodeError gained the same either-shape tolerance. The wrapper is preferred whenever present, so decode is byte-identical against every daemon shipping today; the fallback cannot fire until a flag-dayed daemon exists. Tolerance is scoped to the envMessage/envData/DecodeError call sites only, never envNone, so a genuine top-level message/data field on an already-raw response is never routed through the peek. Covered by pkg/apiclient/envelope_internal_test.go (both shapes, both keys) and end-to-end in client_test.go.

Step 2: daemon + CLI flag-day. OWED. Replace WriteMessage/WriteDoc at all 57 sites with WriteJSON; delete the two envelope structs; update the CLI and x9s decoders to read the un-nested body (delete the Message/Data wrapper level from each daemon_client.go / types.go). Land ce first, then shc with the pin bumped. Because shc ships with shcd, the CLI half lands in one deb with no partial state. This is mechanical breadth, not depth.

Step 3: drop the shim. OWED, optional. Once the tolerant provider is the enforced minimum pin across all estates, delete the peek fallback so the decoder is single-dialect again. Not correctness; the shim is harmless if left.

This ordering respects the crux: at no point does a pinned provider meet a daemon whose dialect it cannot parse.

The x9s effort (“catch shc-x9s up to the current wire: column roles and relations”, e77c6ab08) modernizes a different wire: the k9s provider protobuf that cmd/shc-x9s speaks to the k9s TUI host (ResourceType, ListReq, Column.Role, Relations). That contract has no relationship to the daemon’s HTTP success/error envelopes. x9s is a consumer of the HTTP dialect described here (it decodes the same {"message": …} bodies the CLI does) so the flag-day in step 2 includes its decoders. Nothing about the x9s catch-up decided, replaced, or obsoleted this proposal.

What breaks if done wrong: every CLI command (each daemon_client.go decodes a specific envelope; a missed one silently returns a zero-value struct, not an error, the nastiest failure mode, so decode changes must be compile-enforced by deleting the wrapper type, not by making it optional) and every provider resource/data-source. The double-wrap sites and the pagination-bearing list structs are the two places a careless “unwrap .data” loses data; the target rule (“un-nest the object,” not “read .data”) is written to forbid exactly that.

  • Unit, server: respond_test.go currently PINS the envelopes (it asserts WriteMessage wraps in {"message": …}). Step 2 inverts it: assert WriteJSON is the only writer, plus a grep-guard for zero non-test hits on WriteMessage/WriteDoc across both repos; an shc-only scan would miss 42 of the 54 sites, and a ce-only scan would miss 12.
  • Unit, provider: table-test UnwrapEnvelope and DecodeError against both a wrapped and a raw body for each envelope, the dual-dialect proof. Already in place from step 1; keep it through step 2 and delete it with the shim at step 3.
  • Unit, CLI: each touched daemon_client.go gets a fixture of the new raw body; the compiler catches the rest because the wrapper field is deleted.
  • e2e/bats: the app suites exercise real shc commands against a live daemon; they pass unchanged only if the flag-day deb is internally consistent, so they ARE the CLI cross-check.
  • Provider acceptance: run the suite TWICE during the transition: the tolerant provider against an old (wrapped) daemon AND against a new (raw) one. That is the executable form of the crux verdict.
  • Provider shim (step 1): S, done.
  • CLI + server flag-day (step 2): M, mechanical but broad (57 server sites across two repos + ~115 decoder files); depth is trivial, breadth is the cost. Best done module-by-module with the compiler as the guardrail.
  • Shim removal (step 3): S, optional.
  • Error-envelope change: S if folded in (two choke points), independently sequenceable.
  • No route-path renames. /api/method/* and /api/resource/* keep their paths even though method/resource are Frappe-isms; renaming URLs is a separate, larger break (every generated path string, every provider endpoint, every bats invocation) with no wire-dialect payoff. Paths stay; only bodies change.
  • No auth change. Bearer/UDS/ExtractAuth, the ?token= query fallback, and the 204 + X-SHC-Warning-Code warning path are untouched.
  • No streaming change. invokeStream (log tails, PEM/file downloads) already returns the raw body and never wrapped, out of scope.
  • The error-envelope swap is optional. It can ride this sweep or be deferred; the success-envelope removal stands on its own and carries the blast radius. Do not let the smaller, independent error change gate the larger one.
  • No new response fields or pagination redesign. List responses keep their existing next_cursor/has_more/total_count shape; they merely stop being nested inside a wrapper.