DNS Provider Sync and Traefik ACME Architecture
Layer 10. Edge. Edge
This document describes the architecture for DNS provider integration and Traefik ACME certificate management in SHC.
Overview
Section titled “Overview”SHC’s DNS integration consists of three main components:
- DNS Provider Layer: Manages DNS records via provider APIs
- DNS Service: High-level operations for deployment DNS management
- Traefik ACME Integration: Certificate issuance via Let’s Encrypt
Architecture Diagram
Section titled “Architecture Diagram”flowchart TB CLI["CLI/API"] --> SVC["DNS Service"] SVC --> FACTORY["DNS Provider Factory"] SVC --> RENDERER["Traefik ACME Renderer"] SVC --> RECMGR["Record Manager"] FACTORY --> CF["CF"] FACTORY --> R53["R53"] FACTORY --> HZ["HZ"] FACTORY --> PD["PD"] RENDERER --> TRAEFIK["Traefik"] RECMGR --> DNSAPI["DNS API"]DNS Provider Layer
Section titled “DNS Provider Layer”Provider Interface
Section titled “Provider Interface”All DNS providers implement the DNSProvider interface:
type DNSProvider interface { GetZoneID(ctx context.Context, domain string) (string, error) CreateRecord(ctx context.Context, record DNSRecord) (DNSRecord, error) UpdateRecord(ctx context.Context, record DNSRecord) (DNSRecord, error) DeleteRecord(ctx context.Context, record DNSRecord) error GetRecord(ctx context.Context, zoneID, name, recordType string) (*DNSRecord, error) ListRecords(ctx context.Context, zoneID string, opts ...ListOption) ([]DNSRecord, error) CreateOrUpdateRecord(ctx context.Context, record DNSRecord) (DNSRecord, error) // Idempotent GetTraefikEnv() map[string]string // For DNS-01 challenge}Provider Factory
Section titled “Provider Factory”The factory follows the same pattern as VaultProviderFactory:
// Resolution order:// 1. provider_override (CLI --dns-provider)// 2. dns.provider (explicit config)// 3. First key in dns.providers// 4. Error if none configuredSupported Providers
Section titled “Supported Providers”The “Traefik Env Vars” column is the exact key set the provider’s
TraefikEnv() returns for a fully-populated config; it is what lego reads
inside the Traefik container to run the DNS-01 challenge. A provider whose
config omits an optional field simply emits fewer keys; nothing here is
advisory.
tests/unit/modules/dns/doc_traefik_env_test.go asserts this table against
the ce build actually pinned in go.mod, so a provider that changes its env
contract reds the guard instead of rotting the table.
| Provider | Required Config | Traefik Env Vars |
|---|---|---|
| Cloudflare | token | CF_DNS_API_TOKEN |
| DigitalOcean | token | DO_AUTH_TOKEN |
| Exoscale | api_key, api_secret | EXOSCALE_API_KEY, EXOSCALE_API_SECRET, EXOSCALE_API_ZONE, EXOSCALE_ENDPOINT |
| Hetzner | token | HETZNER_API_TOKEN, HETZNER_PROPAGATION_TIMEOUT |
| Linode | token | LINODE_TOKEN |
| Namecheap | api_user, api_key | NAMECHEAP_API_KEY, NAMECHEAP_API_USER |
| OVH | application_key, application_secret, consumer_key | OVH_APPLICATION_KEY, OVH_APPLICATION_SECRET, OVH_CONSUMER_KEY, OVH_ENDPOINT |
| PowerDNS | api_key, endpoint | PDNS_API_KEY, PDNS_API_URL |
| Scaleway | secret_key | SCW_SECRET_KEY |
| Vultr | api_key | VULTR_API_KEY |
★ HETZNER_API_KEY is the wrong variable and it fails SILENTLY. This
table named it until 2026-08-07, and lego honours it, by routing the
challenge down its legacy client, whose base URL is the retired
https://dns.hetzner.com. That host now answers 301 → console.hetzner.com,
so a client following redirects gets HTTP 200 carrying an HTML page: a
success code with no data, which is the worst shape an API failure can take.
HETZNER_API_TOKEN selects lego’s hetznerv1 client against
https://api.hetzner.cloud/v1, which is where Hetzner DNS actually lives, and
is what hetzner.Provider.TraefikEnv() emits. One Hetzner Cloud token covers
both the Cloud API and DNS. (lego v5 keeps HETZNER_API_KEY only as a
deprecated alias and prefers HETZNER_API_TOKEN when both are set, so the
damage lands on an operator who sets the old name alone, from a table like
this one.)
Measured by hand, not guarded: their constructors take a context and
walk an SDK credential chain, which a unit test must not do:
| Provider | Required Config | Traefik Env Vars |
|---|---|---|
| Route53 | region | AWS_REGION, plus AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_HOSTED_ZONE_ID when set |
| Azure | subscription_id, resource_group | AZURE_SUBSCRIPTION_ID, AZURE_RESOURCE_GROUP, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_TENANT_ID |
| GCP | project | GCE_PROJECT, GCE_SERVICE_ACCOUNT, GOOGLE_APPLICATION_CREDENTIALS |
| Designate | OpenStack auth | OS_AUTH_URL, OS_USERNAME, OS_PASSWORD, OS_USER_DOMAIN_NAME, OS_PROJECT_DOMAIN_NAME, OS_PROJECT_NAME, OS_PROJECT_ID, OS_REGION_NAME |
Providers that emit NO Traefik env vars, by design: bind (RFC 2136
dynamic update), pfsense (Unbound), local and cloud. They are not lego
DNS-01 solvers (SHC writes their records itself), so an empty TraefikEnv()
is the correct answer, not a missing one.
DNS Record Management
Section titled “DNS Record Management”Record Type Selection
Section titled “Record Type Selection”Based on target host resolution:
| Target | Record Type |
|---|---|
| 192.168.1.1 | A |
| 2001:db8::1 | AAAA |
| example.com | CNAME |
Service.ResolveTarget is the only record-type selector on the write path,
and this table is its entire range: a target is an IPv4 literal, an IPv6
literal, or a name, and there is no fourth answer. doc_traefik_env_test.go
asserts this table against the ce build pinned in go.mod too, so it cannot
drift from the selector any more than the provider table can drift from
TraefikEnv().
A host changing record type
Section titled “A host changing record type”A CNAME shares its name with nothing, and every public backend refuses one beside an A or AAAA. So when a host SHC owns moves between a CNAME and addresses, the sync changes the type before it reconciles each type, in one of two ways, picked by what the provider declares:
| Provider | How the type changes |
|---|---|
| cloudflare, route53, gcp, powerdns, bind, namecheap | one write that removes the old rows and adds the new one, applied as a whole (batch, change set, multi-rrset PATCH, rfc2136 UPDATE, setHosts); the name answers throughout |
| the others | the old rows are deleted and the new one created straight after; if the create fails the old rows are written back |
A failed change says so: X502870DNSTYR when the old record still answers,
X502871DNSTYL when the restore failed too. A host with no ownership claim is
never changed this way; its rows wait for the claim like any other prune.
Where the ownership claim lives
Section titled “Where the ownership claim lives”SHC proves a host is its own with a TXT at _shc-owner.<host> reading
heritage=shc,shc/owner=<dns.owner_id> (plus shc/generation=… and
shc/pinned=1 when they apply). It sits one label down because a CNAME
shares its name with nothing: Route53, Cloud DNS, PowerDNS and Cloudflare
refuse a TXT beside one and BIND drops it, so a claim at the host itself would
block every host whose target is a hostname (SHC-1726).
- The claim is read and written only at
_shc-owner.<host>. Aheritage=shcTXT at the host itself is not a claim: sync treats the host as unowned, delete refuses it (X403041DNSUNO), and the reaper never sees it. - The claim is written before the host’s records and removed after them, so
no record of ours ever stands without the claim that proves it ours. An
explicit create (
POST /api/dns/record) whose record write fails takes its fresh claim back out. Under a wildcard the order flips (next section). - A wildcard route host’s claim is its domain’s (next sections). A
*with no plain domain of two labels under it has no claim name and is refused withX409872DNSOWW. A claim name that holds a CNAME, or that the provider answers for from a wildcard, is refused withX409873DNSOWC; remove the record there before SHC manages the host. - The
dnssocket never writes a_shc-ownername.
Public deployment readiness
Section titled “Public deployment readiness”Install and update collect the successfully published public-plane hosts in a deployment-scoped context. After L7 and L4 publication and their deferred commits return, a single readiness barrier runs outside the zone locks. Private planes, unmanaged names and refused writes collect nothing. Background reconciliation does not create a collector or wait for propagation.
Authority discovery starts from literal root-server IPv4 addresses and follows
referrals with recursion disabled. Glue supplies nameserver addresses;
missing glue is resolved through the same iterative path. Cloudflare can supply
the zone name and name_servers through GET /zones/{id}, including pinned
zone IDs, but its nameserver addresses still come from iterative DNS. No system
or recursive resolver participates. Every DNS exchange targets a literal IP
on port 53, with a TCP retry when UDP is truncated.
The first pass queries all authoritative nameserver addresses concurrently. Each must return an authoritative NOERROR answer with every expected A or AAAA address, or the expected CNAME. A proxied Cloudflare row instead requires a non-empty authoritative answer of its record type because Cloudflare returns proxy addresses rather than the origin. NXDOMAIN, NODATA, wrong targets and non-authoritative replies remain pending. Already answering names, including covering wildcards and unchanged rows, need no polling delay. Pending servers are polled every two seconds.
Discovery and polling share a 90-second deadline. On expiry the deploy retains
exit code 0 and records one coded warning per pending host on its verdict.
X200145DNSRDY names the host and nameservers still pending. X200146DNSAUT
names the host and zone when authority discovery could not complete. The
containers, provider rows and routes remain deployed. There is no readiness
progress output on success. This verifies the authoritative endpoints reached
from the deployment node; it cannot clear answers already cached by clients.
One wildcard per declared domain
Section titled “One wildcard per declared domain”shc deploy sends a DNS base as the install or update’s domain field:
the zone the first hostname lives in, with the environment’s prefix
(staging.<zone> or <slug>.review.<zone>, and the zone itself for prod).
The deployment records it. Dev and test hosts land inside that base
(app.<zone> is app.dev.<zone>), so a deployfile deploy’s hosts directly
under it share the wildcard below. Review hosts keep <slug>.review.<host>
and publish per-host records. The daemon’s wildcard rule is unchanged.
A plain shc install or shc update states no base, and an update that
states nothing keeps what is recorded. When hosts directly under the
recorded base share one address, it holds *.<base> and one claim at
_shc-owner._wildcard.<base>. A host the wildcard answers for holds no record
of its own (SHC-3055). A Cloudflare Free zone holds 200 records, and an A
plus a claim per host filled one at about a hundred hosts. Every other host
keeps its own record and claim. Only a base a deployment records gets a
wildcard, and never the zone’s apex: *.<zone> would answer for every name
in the zone.
- Only the deployment lifecycle writes it, only through a provider that
declares it serves wildcards (cloudflare today), and only for an address:
a CNAME wildcard would be followed by every
_acme-challenge.<host>lookup. - The first host under a domain writes the claim, reads it back (beside another owner’s claim it stands down; of two of ours, one stays), then writes the wildcard. A later host with the same address writes nothing, and one that still holds its own record and claim from before has both removed, claim first, which frees the zone.
- A host with another address keeps its own record and claim, which win over the wildcard. The wildcard moves to a new address only once no desired host resolves through it, and the hosts whose records then equal it are taken back under it. With the desired-host set unreadable it never moves.
- A wildcard with no claim of ours, or with another owner’s claim beside ours, is never written or removed; the hosts under it publish their own records. Neither is a host record another owner holds, or one the explicit record surface holds.
- A zone that refuses the claim or the wildcard leaves the host on its own records exactly as before.
- A wildcard does not answer for a name that exists, and a name exists as soon as anything stands at it or under it. So a host is left to the wildcard only while nothing but its own records and claim stands there, and before anything is written at or under a host the wildcard answers for (another host, a domain, a mail record, a DNS-01 challenge, an explicit record) that host gets its own record and then its claim. A challenge’s pin is taken back when the order ends, while the wildcard still stands and publishes what the pinned host does; otherwise the host keeps its own records. When a host’s domain cannot be read, or its zone is not known, the covered host above it is pinned the same way before anything is written under it.
- Deleting the last desired host that resolves through the wildcard removes the wildcard, then the claim, once two reads agree no wildcard row is left. The same holds when a sync finds no desired host relying on the wildcard any more, for instance once no live deployment declares the domain: the wildcard goes in that pass. The desired hosts are read again right before the delete. A wildcard that is not ours is never weighed for it: the release reads the domain’s claim at its exact name and lists the domain only under a wildcard of ours. The hosts an uninstall removes together do not hold it up for each other. The reaper removes a wildcard no desired host sits under, and the teardown release removes it with everything else.
A wildcard route host
Section titled “A wildcard route host”A deployment may route *.<domain> itself (--expose '*.pages.example.com=gitlab:8090',
see published ports).
Its record is the *.<domain> a declared domain gets, under the same claim at
_shc-owner._wildcard.<domain>, so the two are one record and never rivals
(SHC-3110).
- It is published only through a provider that serves wildcards
(
X409966DNSWNSnames the provider that does not), never at the zone’s apex (X409967DNSWAP), and only for addresses (X409968DNSWAD). A private plane that refuses it beside a public one warns and the public plane carries the host, as any private plane’s failure does. - While a deployment routes the wildcard, no declared host rewrites or releases it. A host on another address keeps its own record.
- Routing the wildcard to another address first gives every desired host that resolves through it a record of its own at the address it resolves to now, then its claim.
- Deleting the route host removes the wildcard, then the claim, unless a desired host still resolves through it; then both stay as the domain’s. The reaper keeps the wildcard while the route host is desired.
★ The managed record set is A/AAAA/CNAME; mail DNS is deliberately outside it
Section titled “★ The managed record set is A/AAAA/CNAME; mail DNS is deliberately outside it”RecordType declares six kinds (A, AAAA, CNAME, TXT, MX, NS) because the
Provider boundary has to carry back whatever a zone walk returns. The
managed set (what the ownership registry claims, what SyncRecordsForHost
reconciles, and what the orphan reap and the teardown release sweep) is the
hard-coded triple {A, AAAA, CNAME}, repeated in five loops across
service.go and reap.go.
| Record kind | SHC’s relationship to it |
|---|---|
| A / AAAA / CNAME at a managed host | published, ownership-stamped, reconciled, swept |
the ownership TXT at _shc-owner.<host> (heritage=shc,shc/owner=…) | the marker itself |
the _acme-challenge TXT | written and removed by the DNS-01 solver |
| MX, SPF, DKIM, DMARC, the apex, NS | never written, never claimed, structurally invisible |
RecordTypeMX and RecordTypeNS are re-exported by factory.go and honoured
by several provider backends on the read/encode path, but no service-level
call can produce one. ResolveTarget cannot return them. SyncRecordsForHost
takes a typed TargetSpec, so a caller may pass an MX target, and it is
collected into the desired map and then silently dropped, because the reconcile
loop that follows visits only the managed triple. CreateRecordForHost takes a
bare target string and routes through ResolveTarget. So there is no API
(Go, HTTP, or CLI) that writes an MX, an SPF, a DMARC, or an application’s
DKIM TXT.
★ This is a load-bearing design property, not an oversight. The reaper’s
whole safety argument is that it “never looks at a record it did not provably
write”: its candidate set is exactly the hosts carrying this cluster’s
ownership TXT, and reap.go names “the apex, mail/MX, SPF/DKIM/DMARC TXT” as
the canonical examples of what that invisibility protects. Mail records are
safe from the sweeps because they are unclaimed. Publishing them under
heritage=shc would be the same act as making them deletable by the next
reap pass whose desired-host set does not name them, pointing the one
sweep that has already leaked records (§ Absence is not failure) at the one
record class whose deletion is silent, total, and invisible until mail stops
arriving.
Publishing mail DNS is therefore a new capability with an unanswered ownership
question, not a missing surface over an existing one. The open question (and
the shc dns verb proposed to close gap-inventory G33/G43/G51) is written up
in docs/proposals/2026-08-07-shc-dns-verb-doctrine-question.md.
★ Reading the ownership registry: shc doctor, not shc dns
Section titled “★ Reading the ownership registry: shc doctor, not shc dns”Records are lifecycle-managed and there is deliberately no shc dns CLI
(modules/dns/register.go, modules/dns/router.go, and the
removedTopLevelCommands pin in tests/unit/cli/cli_test.go all say so). That
ruling is about mutation: operator record CRUD. Asking what the estate
already owns is not mutation, and it arrives as a doctor diagnosis instead:
dns.record-ownership (ce modules/dns/doctor.go) runs the same
cluster-wide sweep the orphan reaper does (the connection walk, the
cross-tenant provider resolution, the inline-provider arm) in a read-only
audit mode that adds one zone-wide enumeration of the managed types, so a
claim can be compared against the rows standing at it. It reports:
| State | Tier | What it means |
|---|---|---|
| orphaned claims | warning | this cluster’s claim on a host its desired-host set does not name. The 2026-07-31 sweep found 44 of these, two resolving to addresses the cloud had recycled to third parties |
| dark hosts | warning | claimed, intended, and no A/AAAA/CNAME at all; the record was removed outside SHC and nothing republishes it until the deployment is rendered again |
| foreign-claimed hosts | warning | a host this cluster intends to publish whose records carry another owner’s claim: every sync skips it (X403040DNSOWN) and it never converges |
| unadopted predecessor claims | warning | a previous build of this cluster fqdn published them and the desired set cannot prove them ours (X403044DNSRPL); no sweep will ever settle them |
| unreachable zones | warning | a zone that could not be enumerated. Its silence is not an all-clear |
| pinned claims | info | shc/pinned=1, exempt from the orphan reaper permanently, so they survive an uninstall |
| unclaimed intended hosts | info | live records carrying no ownership TXT: delete refuses them today (X403041DNSUNO), and the next managed sync self-adopts the claim |
| no zone reachable | info | nothing was established about this estate at all |
It prescribes nothing and --fix cannot touch it. Every remedy it can
name mutates a live zone or re-renders a running deployment, and the one sweep that
could be automated (POST /api/dns/reap?apply=true) has already leaked in
production. The registry entry carries Name and Build only; ce’s
modules/dns/doctor_test.go pins that, and shc’s
tests/unit/modules/dns/audit_test.go proves the no-write property at the
provider boundary: zero creates, zero updates, zero deletes, and a
byte-identical zone, over a fixture that deliberately includes the one claim a
reap apply would rewrite.
It inherits the mail-record invisibility above rather than working around it: the apex, MX, SPF, DKIM and DMARC carry no claim, so they appear in no column.
Idempotency
Section titled “Idempotency”The create_or_update_record method ensures idempotency:
- Get existing record for name+type
- If exists with identical content/TTL: no-op
- If exists with different content: update
- If not exists: create
Traefik runs no ACME client and holds no provider credential. The daemon’s
ACME source (ce modules/acme) orders each certificate once and every
Traefik replica serves it. See Networking.
- HTTP-01: every node’s daemon answers
/.well-known/acme-challenge/from the replicated store. - DNS-01: the daemon publishes the challenge TXT record through the public DNS connection that owns the zone.
HTTP-01 vs DNS-01
Section titled “HTTP-01 vs DNS-01”| Aspect | HTTP-01 | DNS-01 |
|---|---|---|
| Port 80 required | Yes | No |
| Works for internal | No | Yes |
| Wildcard certs | No | Yes |
| DNS provider needed | No | Yes |
Deployment DNS Configuration
Section titled “Deployment DNS Configuration”Each deployment can have DNS configuration:
type DeploymentDNSConfig struct { Hostname string Provider string // Override (optional) TLSMode TLSMode // NONE, HTTP, DNS CleanupOnUninstall bool}Security Considerations
Section titled “Security Considerations”- Credential Storage: Use Vault references for provider credentials
- Minimal Exposure: Traefik only receives credentials for configured providers
- Scoped Config: Provider selection follows the scope ladder. Deployment is the narrowest tier; stack and environment are axes (
ce core/scope/scope.go).
Error Handling
Section titled “Error Handling”| Error | Description | Resolution |
|---|---|---|
| X503031DNSCFG | No DNS provider configured | Add dns.providers config |
| X400922DNSUNK | Unknown provider | Use supported provider name |
| X400926DNSREQ | Missing required config | Add required provider config |
| X400032DNSZON | Zone not found | Verify domain is in provider |
| X404047DNSNFD | Provider reports the zone/record absent | Nothing to do: the only error a caller may read as absence |
| X503030DNSAPI | Provider API error | Check credentials and connectivity |
Absence is not failure. X404047DNSNFD is the provider’s NOT-FOUND
answer and the ONLY error any caller may turn into a silent no-op, a
(nil, nil) from GetRecord, a (false, nil) from DeleteRecord, a
continue in a zone walk. Every other failure (5xx, 401/403, rate limit,
transport failure, cancelled context) propagates as X503030DNSAPI.
The distinction is load-bearing at teardown, not cosmetic. The
pre-destroy release asks “does this record exist?” and treats “no” as
“nothing to clean up”; before the split, providers matched absence
against the GENERIC API error, so a transient failure answered “no” and
the release reported a clean teardown over records that were still live.
The 2026-07-31 cross-provider sweep found 44 heritage=shc records that
had outlived their estates, two of them resolving to IPs the cloud had
already recycled to third parties, subdomain-takeover material, publicly
trusted certificate issuance included. A release that REACHED nothing
must stay distinguishable from a release that could not LOOK, which is
what ReapReport.ZoneErrors and X200905DNSRLZ report.
Operational Notes
Section titled “Operational Notes”- Idempotent Operations: DNS record creation is safe to retry
- Cleanup Policy: uninstall releases the deployment’s managed-DNS records by default.
KeepDNSleaves them when a same-key relocation will register them again (ce daemon/services_seams_stack.go).