Skip to content
SHC Docs

DNS Provider Sync and Traefik ACME Architecture

Layer 10. Edge. Edge

This document describes the architecture for DNS provider integration and Traefik ACME certificate management in SHC.

SHC’s DNS integration consists of three main components:

  1. DNS Provider Layer: Manages DNS records via provider APIs
  2. DNS Service: High-level operations for deployment DNS management
  3. Traefik ACME Integration: Certificate issuance via Let’s Encrypt
flowchart TB
CLI["CLI/API"] --> SVC["DNS Service"]
SVC --> FACTORY["DNS Provider Factory"]
SVC --> RENDERER["Traefik ACME Renderer"]
SVC --> RECMGR["Record Manager"]
FACTORY --> CF["CF"]
FACTORY --> R53["R53"]
FACTORY --> HZ["HZ"]
FACTORY --> PD["PD"]
RENDERER --> TRAEFIK["Traefik"]
RECMGR --> DNSAPI["DNS API"]

All DNS providers implement the DNSProvider interface:

type DNSProvider interface {
GetZoneID(ctx context.Context, domain string) (string, error)
CreateRecord(ctx context.Context, record DNSRecord) (DNSRecord, error)
UpdateRecord(ctx context.Context, record DNSRecord) (DNSRecord, error)
DeleteRecord(ctx context.Context, record DNSRecord) error
GetRecord(ctx context.Context, zoneID, name, recordType string) (*DNSRecord, error)
ListRecords(ctx context.Context, zoneID string, opts ...ListOption) ([]DNSRecord, error)
CreateOrUpdateRecord(ctx context.Context, record DNSRecord) (DNSRecord, error) // Idempotent
GetTraefikEnv() map[string]string // For DNS-01 challenge
}

The factory follows the same pattern as VaultProviderFactory:

// Resolution order:
// 1. provider_override (CLI --dns-provider)
// 2. dns.provider (explicit config)
// 3. First key in dns.providers
// 4. Error if none configured

The “Traefik Env Vars” column is the exact key set the provider’s TraefikEnv() returns for a fully-populated config; it is what lego reads inside the Traefik container to run the DNS-01 challenge. A provider whose config omits an optional field simply emits fewer keys; nothing here is advisory.

tests/unit/modules/dns/doc_traefik_env_test.go asserts this table against the ce build actually pinned in go.mod, so a provider that changes its env contract reds the guard instead of rotting the table.

ProviderRequired ConfigTraefik Env Vars
CloudflaretokenCF_DNS_API_TOKEN
DigitalOceantokenDO_AUTH_TOKEN
Exoscaleapi_key, api_secretEXOSCALE_API_KEY, EXOSCALE_API_SECRET, EXOSCALE_API_ZONE, EXOSCALE_ENDPOINT
HetznertokenHETZNER_API_TOKEN, HETZNER_PROPAGATION_TIMEOUT
LinodetokenLINODE_TOKEN
Namecheapapi_user, api_keyNAMECHEAP_API_KEY, NAMECHEAP_API_USER
OVHapplication_key, application_secret, consumer_keyOVH_APPLICATION_KEY, OVH_APPLICATION_SECRET, OVH_CONSUMER_KEY, OVH_ENDPOINT
PowerDNSapi_key, endpointPDNS_API_KEY, PDNS_API_URL
Scalewaysecret_keySCW_SECRET_KEY
Vultrapi_keyVULTR_API_KEY

★ HETZNER_API_KEY is the wrong variable and it fails SILENTLY. This table named it until 2026-08-07, and lego honours it, by routing the challenge down its legacy client, whose base URL is the retired https://dns.hetzner.com. That host now answers 301 → console.hetzner.com, so a client following redirects gets HTTP 200 carrying an HTML page: a success code with no data, which is the worst shape an API failure can take. HETZNER_API_TOKEN selects lego’s hetznerv1 client against https://api.hetzner.cloud/v1, which is where Hetzner DNS actually lives, and is what hetzner.Provider.TraefikEnv() emits. One Hetzner Cloud token covers both the Cloud API and DNS. (lego v5 keeps HETZNER_API_KEY only as a deprecated alias and prefers HETZNER_API_TOKEN when both are set, so the damage lands on an operator who sets the old name alone, from a table like this one.)

Measured by hand, not guarded: their constructors take a context and walk an SDK credential chain, which a unit test must not do:

ProviderRequired ConfigTraefik Env Vars
Route53regionAWS_REGION, plus AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_HOSTED_ZONE_ID when set
Azuresubscription_id, resource_groupAZURE_SUBSCRIPTION_ID, AZURE_RESOURCE_GROUP, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_TENANT_ID
GCPprojectGCE_PROJECT, GCE_SERVICE_ACCOUNT, GOOGLE_APPLICATION_CREDENTIALS
DesignateOpenStack authOS_AUTH_URL, OS_USERNAME, OS_PASSWORD, OS_USER_DOMAIN_NAME, OS_PROJECT_DOMAIN_NAME, OS_PROJECT_NAME, OS_PROJECT_ID, OS_REGION_NAME

Providers that emit NO Traefik env vars, by design: bind (RFC 2136 dynamic update), pfsense (Unbound), local and cloud. They are not lego DNS-01 solvers (SHC writes their records itself), so an empty TraefikEnv() is the correct answer, not a missing one.

Based on target host resolution:

TargetRecord Type
192.168.1.1A
2001:db8::1AAAA
example.comCNAME

Service.ResolveTarget is the only record-type selector on the write path, and this table is its entire range: a target is an IPv4 literal, an IPv6 literal, or a name, and there is no fourth answer. doc_traefik_env_test.go asserts this table against the ce build pinned in go.mod too, so it cannot drift from the selector any more than the provider table can drift from TraefikEnv().

A CNAME shares its name with nothing, and every public backend refuses one beside an A or AAAA. So when a host SHC owns moves between a CNAME and addresses, the sync changes the type before it reconciles each type, in one of two ways, picked by what the provider declares:

ProviderHow the type changes
cloudflare, route53, gcp, powerdns, bind, namecheapone write that removes the old rows and adds the new one, applied as a whole (batch, change set, multi-rrset PATCH, rfc2136 UPDATE, setHosts); the name answers throughout
the othersthe old rows are deleted and the new one created straight after; if the create fails the old rows are written back

A failed change says so: X502870DNSTYR when the old record still answers, X502871DNSTYL when the restore failed too. A host with no ownership claim is never changed this way; its rows wait for the claim like any other prune.

SHC proves a host is its own with a TXT at _shc-owner.<host> reading heritage=shc,shc/owner=<dns.owner_id> (plus shc/generation=… and shc/pinned=1 when they apply). It sits one label down because a CNAME shares its name with nothing: Route53, Cloud DNS, PowerDNS and Cloudflare refuse a TXT beside one and BIND drops it, so a claim at the host itself would block every host whose target is a hostname (SHC-1726).

  • The claim is read and written only at _shc-owner.<host>. A heritage=shc TXT at the host itself is not a claim: sync treats the host as unowned, delete refuses it (X403041DNSUNO), and the reaper never sees it.
  • The claim is written before the host’s records and removed after them, so no record of ours ever stands without the claim that proves it ours. An explicit create (POST /api/dns/record) whose record write fails takes its fresh claim back out. Under a wildcard the order flips (next section).
  • A wildcard route host’s claim is its domain’s (next sections). A * with no plain domain of two labels under it has no claim name and is refused with X409872DNSOWW. A claim name that holds a CNAME, or that the provider answers for from a wildcard, is refused with X409873DNSOWC; remove the record there before SHC manages the host.
  • The dns socket never writes a _shc-owner name.

Install and update collect the successfully published public-plane hosts in a deployment-scoped context. After L7 and L4 publication and their deferred commits return, a single readiness barrier runs outside the zone locks. Private planes, unmanaged names and refused writes collect nothing. Background reconciliation does not create a collector or wait for propagation.

Authority discovery starts from literal root-server IPv4 addresses and follows referrals with recursion disabled. Glue supplies nameserver addresses; missing glue is resolved through the same iterative path. Cloudflare can supply the zone name and name_servers through GET /zones/{id}, including pinned zone IDs, but its nameserver addresses still come from iterative DNS. No system or recursive resolver participates. Every DNS exchange targets a literal IP on port 53, with a TCP retry when UDP is truncated.

The first pass queries all authoritative nameserver addresses concurrently. Each must return an authoritative NOERROR answer with every expected A or AAAA address, or the expected CNAME. A proxied Cloudflare row instead requires a non-empty authoritative answer of its record type because Cloudflare returns proxy addresses rather than the origin. NXDOMAIN, NODATA, wrong targets and non-authoritative replies remain pending. Already answering names, including covering wildcards and unchanged rows, need no polling delay. Pending servers are polled every two seconds.

Discovery and polling share a 90-second deadline. On expiry the deploy retains exit code 0 and records one coded warning per pending host on its verdict. X200145DNSRDY names the host and nameservers still pending. X200146DNSAUT names the host and zone when authority discovery could not complete. The containers, provider rows and routes remain deployed. There is no readiness progress output on success. This verifies the authoritative endpoints reached from the deployment node; it cannot clear answers already cached by clients.

shc deploy sends a DNS base as the install or update’s domain field: the zone the first hostname lives in, with the environment’s prefix (staging.<zone> or <slug>.review.<zone>, and the zone itself for prod). The deployment records it. Dev and test hosts land inside that base (app.<zone> is app.dev.<zone>), so a deployfile deploy’s hosts directly under it share the wildcard below. Review hosts keep <slug>.review.<host> and publish per-host records. The daemon’s wildcard rule is unchanged.

A plain shc install or shc update states no base, and an update that states nothing keeps what is recorded. When hosts directly under the recorded base share one address, it holds *.<base> and one claim at _shc-owner._wildcard.<base>. A host the wildcard answers for holds no record of its own (SHC-3055). A Cloudflare Free zone holds 200 records, and an A plus a claim per host filled one at about a hundred hosts. Every other host keeps its own record and claim. Only a base a deployment records gets a wildcard, and never the zone’s apex: *.<zone> would answer for every name in the zone.

  • Only the deployment lifecycle writes it, only through a provider that declares it serves wildcards (cloudflare today), and only for an address: a CNAME wildcard would be followed by every _acme-challenge.<host> lookup.
  • The first host under a domain writes the claim, reads it back (beside another owner’s claim it stands down; of two of ours, one stays), then writes the wildcard. A later host with the same address writes nothing, and one that still holds its own record and claim from before has both removed, claim first, which frees the zone.
  • A host with another address keeps its own record and claim, which win over the wildcard. The wildcard moves to a new address only once no desired host resolves through it, and the hosts whose records then equal it are taken back under it. With the desired-host set unreadable it never moves.
  • A wildcard with no claim of ours, or with another owner’s claim beside ours, is never written or removed; the hosts under it publish their own records. Neither is a host record another owner holds, or one the explicit record surface holds.
  • A zone that refuses the claim or the wildcard leaves the host on its own records exactly as before.
  • A wildcard does not answer for a name that exists, and a name exists as soon as anything stands at it or under it. So a host is left to the wildcard only while nothing but its own records and claim stands there, and before anything is written at or under a host the wildcard answers for (another host, a domain, a mail record, a DNS-01 challenge, an explicit record) that host gets its own record and then its claim. A challenge’s pin is taken back when the order ends, while the wildcard still stands and publishes what the pinned host does; otherwise the host keeps its own records. When a host’s domain cannot be read, or its zone is not known, the covered host above it is pinned the same way before anything is written under it.
  • Deleting the last desired host that resolves through the wildcard removes the wildcard, then the claim, once two reads agree no wildcard row is left. The same holds when a sync finds no desired host relying on the wildcard any more, for instance once no live deployment declares the domain: the wildcard goes in that pass. The desired hosts are read again right before the delete. A wildcard that is not ours is never weighed for it: the release reads the domain’s claim at its exact name and lists the domain only under a wildcard of ours. The hosts an uninstall removes together do not hold it up for each other. The reaper removes a wildcard no desired host sits under, and the teardown release removes it with everything else.

A deployment may route *.<domain> itself (--expose '*.pages.example.com=gitlab:8090', see published ports). Its record is the *.<domain> a declared domain gets, under the same claim at _shc-owner._wildcard.<domain>, so the two are one record and never rivals (SHC-3110).

  • It is published only through a provider that serves wildcards (X409966DNSWNS names the provider that does not), never at the zone’s apex (X409967DNSWAP), and only for addresses (X409968DNSWAD). A private plane that refuses it beside a public one warns and the public plane carries the host, as any private plane’s failure does.
  • While a deployment routes the wildcard, no declared host rewrites or releases it. A host on another address keeps its own record.
  • Routing the wildcard to another address first gives every desired host that resolves through it a record of its own at the address it resolves to now, then its claim.
  • Deleting the route host removes the wildcard, then the claim, unless a desired host still resolves through it; then both stay as the domain’s. The reaper keeps the wildcard while the route host is desired.

★ The managed record set is A/AAAA/CNAME; mail DNS is deliberately outside it

Section titled “★ The managed record set is A/AAAA/CNAME; mail DNS is deliberately outside it”

RecordType declares six kinds (A, AAAA, CNAME, TXT, MX, NS) because the Provider boundary has to carry back whatever a zone walk returns. The managed set (what the ownership registry claims, what SyncRecordsForHost reconciles, and what the orphan reap and the teardown release sweep) is the hard-coded triple {A, AAAA, CNAME}, repeated in five loops across service.go and reap.go.

Record kindSHC’s relationship to it
A / AAAA / CNAME at a managed hostpublished, ownership-stamped, reconciled, swept
the ownership TXT at _shc-owner.<host> (heritage=shc,shc/owner=…)the marker itself
the _acme-challenge TXTwritten and removed by the DNS-01 solver
MX, SPF, DKIM, DMARC, the apex, NSnever written, never claimed, structurally invisible

RecordTypeMX and RecordTypeNS are re-exported by factory.go and honoured by several provider backends on the read/encode path, but no service-level call can produce one. ResolveTarget cannot return them. SyncRecordsForHost takes a typed TargetSpec, so a caller may pass an MX target, and it is collected into the desired map and then silently dropped, because the reconcile loop that follows visits only the managed triple. CreateRecordForHost takes a bare target string and routes through ResolveTarget. So there is no API (Go, HTTP, or CLI) that writes an MX, an SPF, a DMARC, or an application’s DKIM TXT.

★ This is a load-bearing design property, not an oversight. The reaper’s whole safety argument is that it “never looks at a record it did not provably write”: its candidate set is exactly the hosts carrying this cluster’s ownership TXT, and reap.go names “the apex, mail/MX, SPF/DKIM/DMARC TXT” as the canonical examples of what that invisibility protects. Mail records are safe from the sweeps because they are unclaimed. Publishing them under heritage=shc would be the same act as making them deletable by the next reap pass whose desired-host set does not name them, pointing the one sweep that has already leaked records (§ Absence is not failure) at the one record class whose deletion is silent, total, and invisible until mail stops arriving.

Publishing mail DNS is therefore a new capability with an unanswered ownership question, not a missing surface over an existing one. The open question (and the shc dns verb proposed to close gap-inventory G33/G43/G51) is written up in docs/proposals/2026-08-07-shc-dns-verb-doctrine-question.md.

★ Reading the ownership registry: shc doctor, not shc dns

Section titled “★ Reading the ownership registry: shc doctor, not shc dns”

Records are lifecycle-managed and there is deliberately no shc dns CLI (modules/dns/register.go, modules/dns/router.go, and the removedTopLevelCommands pin in tests/unit/cli/cli_test.go all say so). That ruling is about mutation: operator record CRUD. Asking what the estate already owns is not mutation, and it arrives as a doctor diagnosis instead:

dns.record-ownership (ce modules/dns/doctor.go) runs the same cluster-wide sweep the orphan reaper does (the connection walk, the cross-tenant provider resolution, the inline-provider arm) in a read-only audit mode that adds one zone-wide enumeration of the managed types, so a claim can be compared against the rows standing at it. It reports:

StateTierWhat it means
orphaned claimswarningthis cluster’s claim on a host its desired-host set does not name. The 2026-07-31 sweep found 44 of these, two resolving to addresses the cloud had recycled to third parties
dark hostswarningclaimed, intended, and no A/AAAA/CNAME at all; the record was removed outside SHC and nothing republishes it until the deployment is rendered again
foreign-claimed hostswarninga host this cluster intends to publish whose records carry another owner’s claim: every sync skips it (X403040DNSOWN) and it never converges
unadopted predecessor claimswarninga previous build of this cluster fqdn published them and the desired set cannot prove them ours (X403044DNSRPL); no sweep will ever settle them
unreachable zoneswarninga zone that could not be enumerated. Its silence is not an all-clear
pinned claimsinfoshc/pinned=1, exempt from the orphan reaper permanently, so they survive an uninstall
unclaimed intended hostsinfolive records carrying no ownership TXT: delete refuses them today (X403041DNSUNO), and the next managed sync self-adopts the claim
no zone reachableinfonothing was established about this estate at all

It prescribes nothing and --fix cannot touch it. Every remedy it can name mutates a live zone or re-renders a running deployment, and the one sweep that could be automated (POST /api/dns/reap?apply=true) has already leaked in production. The registry entry carries Name and Build only; ce’s modules/dns/doctor_test.go pins that, and shc’s tests/unit/modules/dns/audit_test.go proves the no-write property at the provider boundary: zero creates, zero updates, zero deletes, and a byte-identical zone, over a fixture that deliberately includes the one claim a reap apply would rewrite.

It inherits the mail-record invisibility above rather than working around it: the apex, MX, SPF, DKIM and DMARC carry no claim, so they appear in no column.

The create_or_update_record method ensures idempotency:

  1. Get existing record for name+type
  2. If exists with identical content/TTL: no-op
  3. If exists with different content: update
  4. If not exists: create

Traefik runs no ACME client and holds no provider credential. The daemon’s ACME source (ce modules/acme) orders each certificate once and every Traefik replica serves it. See Networking.

  • HTTP-01: every node’s daemon answers /.well-known/acme-challenge/ from the replicated store.
  • DNS-01: the daemon publishes the challenge TXT record through the public DNS connection that owns the zone.
AspectHTTP-01DNS-01
Port 80 requiredYesNo
Works for internalNoYes
Wildcard certsNoYes
DNS provider neededNoYes

Each deployment can have DNS configuration:

type DeploymentDNSConfig struct {
Hostname string
Provider string // Override (optional)
TLSMode TLSMode // NONE, HTTP, DNS
CleanupOnUninstall bool
}
  1. Credential Storage: Use Vault references for provider credentials
  2. Minimal Exposure: Traefik only receives credentials for configured providers
  3. Scoped Config: Provider selection follows the scope ladder. Deployment is the narrowest tier; stack and environment are axes (ce core/scope/scope.go).
ErrorDescriptionResolution
X503031DNSCFGNo DNS provider configuredAdd dns.providers config
X400922DNSUNKUnknown providerUse supported provider name
X400926DNSREQMissing required configAdd required provider config
X400032DNSZONZone not foundVerify domain is in provider
X404047DNSNFDProvider reports the zone/record absentNothing to do: the only error a caller may read as absence
X503030DNSAPIProvider API errorCheck credentials and connectivity

Absence is not failure. X404047DNSNFD is the provider’s NOT-FOUND answer and the ONLY error any caller may turn into a silent no-op, a (nil, nil) from GetRecord, a (false, nil) from DeleteRecord, a continue in a zone walk. Every other failure (5xx, 401/403, rate limit, transport failure, cancelled context) propagates as X503030DNSAPI.

The distinction is load-bearing at teardown, not cosmetic. The pre-destroy release asks “does this record exist?” and treats “no” as “nothing to clean up”; before the split, providers matched absence against the GENERIC API error, so a transient failure answered “no” and the release reported a clean teardown over records that were still live. The 2026-07-31 cross-provider sweep found 44 heritage=shc records that had outlived their estates, two of them resolving to IPs the cloud had already recycled to third parties, subdomain-takeover material, publicly trusted certificate issuance included. A release that REACHED nothing must stay distinguishable from a release that could not LOOK, which is what ReapReport.ZoneErrors and X200905DNSRLZ report.

  1. Idempotent Operations: DNS record creation is safe to retry
  2. Cleanup Policy: uninstall releases the deployment’s managed-DNS records by default. KeepDNS leaves them when a same-key relocation will register them again (ce daemon/services_seams_stack.go).