Pulp mirror targets
The pulp app takes any number of s3 connections as mirror targets:
shc -t corp update pulp -C mirror.apt-basedlinux=cloudflare -C mirror.bin-basedlinux=cloudflareEach -C mirror.<name>=<connection> is one target. The target name is the
bucket name. Two kinds of connection satisfy it:
| Connection | What happens at bind |
|---|---|
cloudflare (R2) | the daemon creates bucket <name> on R2, mints a token scoped to it, attaches the domain the grant names |
s3 (a held key: MinIO, Hetzner, a pasted R2 key) | nothing is created; bucket <name> must already exist |
Artifact storage is untouched: the s3 plug and vars.s3.* remain the one
STORAGES backend. A mirror target is a copy the fork’s pulp_mirror plugin
(SHC-845) writes on top of it: the chosen distributions as a static apt/file
tree, so public artifacts serve without Pulp in the path.
What a target renders to
Section titled “What a target renders to”Every bound target becomes per-name settings in the shared pulp environment,
which dynaconf folds into settings.MIRROR_TARGETS[<name>]:
PULP_MIRROR_TARGETS__<name>__endpoint https://<account>.r2.cloudflarestorage.com:443PULP_MIRROR_TARGETS__<name>__bucket <name>, or the bucket the mint chosePULP_MIRROR_TARGETS__<name>__region auto / us-east-1 / …PULP_MIRROR_TARGETS__<name>__access_keyPULP_MIRROR_TARGETS__<name>__secret_keyPULP_MIRROR_TARGETS__<name>__path_style trueThe plugin upserts one MirrorTarget row per name from these on start;
which distributions go to which target is the plugin’s own rule set, driven
from its API and the dashboard toggle.
The probe gates the converge
Section titled “The probe gates the converge”Before any pulp process starts, the pulp-mirror-probe start job HeadBuckets
every target with its own key. It is read-only (no mb, rb, cp, sync,
rm, put or delete) and it fails the compose up, and with it the install or update,
loudly:
| Code | Meaning |
|---|---|
X5055MIRNOB | the bucket does not exist at the endpoint; the probe never creates one |
X5056MIRDEN | the key was rejected for that bucket |
X5057MIRUNR | the endpoint did not answer in 30 attempts |
With no target bound the probe prints one line and exits 0, so an install without mirrors is unchanged.
Removing a target
Section titled “Removing a target”Rebind the name to another connection and the old mint is revoked before the
new one is made. Uninstalling the deployment revokes every target’s mint; the
connection can be removed after that (shc connection rm refuses while a
mint is held).
Testing
Section titled “Testing”The app’s bats suite binds an s3 connection to a hosted provider, named by
the SHC_TEST_S3_* variables (-C mirror.based=…), asserts the rendered
settings on pulp-api, and reads the probe’s exit status and log. The probe
command itself runs against a stub
aws for the refusal paths, the same way external-s3’s discovery probe is
tested.
shc’s own artifacts: the rules (SHC-853)
Section titled “shc’s own artifacts: the rules (SHC-853)”Release tags publish the images in scripts/pulp-images.txt through
pulp/shc-images to registry.selfhostedcloud.org/shc/<name>, at both the
release tag and the commit tag. This includes Keycloak behind
images/keycloak-gate. Pulp waits for that job before syncing any image.
Core’s image-publish-record.txt and Keycloak’s
keycloak-publish/image-publish-record.txt are separate artifacts; the
publisher reads both with repeated --record arguments.
scripts/pulp-carried-images.txt lists catalogue images SHC CI does not
build, each with its upstream tag and the digest the catalogue pins.
pulp/shc-images passes this list with --carry, using the job’s Pulp
credential for the same sync and mirror rule, then reading every digest and
tag back anonymously; a tag that no longer names the pinned digest fails the job.
pulp/shc-cli clears PULP_CARRY, so the list runs once per release tag.
The gate pulls the recorded Keycloak digest for linux/amd64 and
linux/arm64, verifies each layer digest, and scans every layer, including
files deleted by later layers. It refuses tls.key paths and PEM private-key
blocks with a body in regular files up to 96 MiB. Every keystore found in a
layer is checked with that platform image’s own keytool, as root. A
PrivateKeyEntry, SecretKeyEntry, unreadable keystore, missing platform,
unreadable layer or digest mismatch fails the gate. Bare PEM headers pass.
Other .key and .pem paths are reported for review. The job log records
rules, layers, paths, sizes and hash prefixes; it never prints key material.
The runner needs both platform emulators and its Docker socket available.
The gate proves the recorded digest. The Pulp remote still syncs tags, so release and commit tags must remain immutable between scanning and syncing. Public digest read-back detects a replacement after mirroring; it does not prevent that race.
Clay’s ruling on shape: one R2 bucket per public hostname, Pulp base paths as
key prefixes, split by protocol and never by product. shc’s artifacts reach
the bucket the same way Based’s do: CI publishes into pulp_file repositories,
pulp_mirror copies the chosen distributions, R2 fronts the name. CI never
writes the bucket.
.gitlab/ci/pulp-publish.yml fills three repositories through
scripts/pulp-publish-shc.py, each behind a guard-free distribution whose base
path is its name. Every job needs PULP_URL and PULP_PASSWORD on the corp
project (SHC-854), and names the environment pulp, the scope
PULP_PASSWORD holds so no other job receives it (SHC-1746). The new line’s tree, shc, is written by a v0.1+ release
tag and nothing else (SHC-2557); shc-binaries and shc-apps are the old
line’s, written from release/v0.0 and v0.0.* tags, and promanager reads them:
| distribution | what CI lands there | from |
|---|---|---|
shc | <version>/ mirrors generic/shc/<version>/ (deb, rpm, apk, darwin tar.gz, checksums) plus that release’s install.sh; RELEASE names the newest release not on BAD_RELEASES; install.sh at the root is the installer of the release RELEASE names | pulp/shc-release (package/release) |
shc | frappe/<app>/<version>/<app>-<sha12>.tar, every bench app tarball frappe/docker/Dockerfile pins, checked against its pinned sha256 | pulp/shc-frappe (frappe/apps) |
shc-binaries | <version>/ mirrors generic/shc/<version>/ (deb, rpm, apk, darwin tar.gz, checksums); latest/ mirrors the moving channel generic/shc/0.0.0-latest/ (shc_latest_linux_<arch>.deb, shc_latest_COMMIT_SHA); RELEASE names the newest release not on BAD_RELEASES | package/release, package/snapshot |
shc-apps | install/latest/install.sh and install/<version>/install.sh; apps/latest/index.yaml and index-infra.yaml with every archive: rewritten to the downloads host; the archives themselves at apps/<name>/<version>/<file> | publish/installer, apps/index |
Every directory carries a SHA256SUMS in sha256sum format.
The target and the rules
Section titled “The target and the rules”The bucket is downloads-selfhostedcloud, bound to the pulp app as
-C mirror.downloads-selfhostedcloud=cloudflare (the daemon creates it on R2
and attaches the hostname the grant names). The plugin upserts the
MirrorTarget from the rendered settings; the fields it carries, as
MirrorTargetSerializer declares them:
{ "name": "downloads-selfhostedcloud", "endpoint_url": "https://<account>.r2.cloudflarestorage.com", "bucket": "downloads-selfhostedcloud", "region": "auto", "path_style": true, "object_acl": "", "key_prefix": "", "credential_name": "downloads-selfhostedcloud", "public_url": "https://downloads.selfhostedcloud.org"}object_acl stays empty on R2 (the bucket policy decides), key_prefix stays
empty so the base path is the first key segment, and public_url is the
hostname the bucket fronts.
One MirrorRule per distribution, all to that target. pulp/shc-release
creates or converges the shc rule itself on every run (it refuses when the
target is not bound); the old line’s two were made by hand. Fields as
MirrorRuleSerializer declares them; a rule sets exactly one of
distribution and base_path_glob, and the plugin refuses both or neither:
curl -u admin -X POST https://artifacts.bitspur.com/pulp/api/v3/mirror/rules/ \ -H 'Content-Type: application/json' -d '{ "name": "shc-binaries", "distribution": "<href of the file distribution shc-binaries>", "enabled": true, "dry_run": false, "public": true, "key_prefix": "", "targets": [{"target": "<href of downloads-selfhostedcloud>"}] }'| rule | selector | target | key_prefix | served at |
|---|---|---|---|---|
shc | distribution = file distribution shc | downloads-selfhostedcloud | "" | https://downloads.selfhostedcloud.org/shc/… |
shc-binaries | distribution = file distribution shc-binaries | downloads-selfhostedcloud | "" | https://downloads.selfhostedcloud.org/shc-binaries/… |
shc-apps | distribution = file distribution shc-apps | downloads-selfhostedcloud | "" | https://downloads.selfhostedcloud.org/shc-apps/… |
nextcloud-apps (later) | base_path_glob = nextcloud-apps* | downloads-selfhostedcloud | nextcloud | https://downloads.selfhostedcloud.org/nextcloud/… |
The edge’s public_url, public and grace_seconds are left null so the
rule’s and the target’s values apply. A key on the bucket is
<target.key_prefix>/<rule.key_prefix>/<base_path>/<relative_path> with
empty prefixes dropped, so shc-binaries/latest/shc_latest_linux_amd64.deb
in Pulp is shc-binaries/latest/shc_latest_linux_amd64.deb on R2. The
nextcloud prefix is the one place a rule prefix is used, because those
distributions carry a protocol of their own under one path.
Runs are queued by the plugin on every completed publication (the CI jobs cut
one each), on a rule save, on the MIRROR_INTERVAL_SECONDS timer, and by
POST /pulp/api/v3/mirror/rules/sync/. Files a publication dropped are
retired for the grace period, then pruned; for shc-apps that is what
retention on the registry turns into on the bucket, because the repository
version is rebuilt to exactly what the index names.
What a consumer pins
Section titled “What a consumer pins”The new line serves every artifact from the shc tree, and its debs from the
Based apt repo as well (SHC-2763): pulp/shc-apt puts each release’s amd64
and arm64 debs in the set based-d13 and the Based suite d13, signed by
that repo’s key, so a box with the Based repo runs apt install shc, or
apt install shc=<ver> to pin. corp’s registry is CI’s internal origin and
answers 403 outside, so nothing a customer reads names it:
| what | URL |
|---|---|
| the installer | https://downloads.selfhostedcloud.org/shc/install.sh |
| the newest release | https://downloads.selfhostedcloud.org/shc/RELEASE |
| a release deb | https://downloads.selfhostedcloud.org/shc/<ver>/shc_<ver>_linux_amd64.deb |
| a release deb by apt | https://apt.basedlinux.com/deb/based, suite d13, component main, package shc |
| a release darwin archive | https://downloads.selfhostedcloud.org/shc/<ver>/shc_<ver>_darwin_arm64.tar.gz |
| its checksums | https://downloads.selfhostedcloud.org/shc/<ver>/shc_<ver>_checksums.txt |
The old line, for the estates that still run it: each registry URL has one mirror URL. The registry stays the durable origin; the mirror is what a user can reach with the estate down.
| what | registry (corp project 22) | mirror |
|---|---|---|
moving deb, deb_url default | …/packages/generic/shc/0.0.0-latest/shc_latest_linux_amd64.deb | https://downloads.selfhostedcloud.org/shc-binaries/latest/shc_latest_linux_amd64.deb |
| a release deb | …/packages/generic/shc/<ver>/shc_<ver>_linux_amd64.deb | https://downloads.selfhostedcloud.org/shc-binaries/<ver>/shc_<ver>_linux_amd64.deb |
| a release darwin archive | …/packages/generic/shc/<ver>/shc_<ver>_darwin_arm64.tar.gz | https://downloads.selfhostedcloud.org/shc-binaries/<ver>/shc_<ver>_darwin_arm64.tar.gz |
| the newest release | the Releases API | https://downloads.selfhostedcloud.org/shc-binaries/RELEASE |
| the installer | …/packages/generic/shc-install/0.0.0-latest/install.sh | https://downloads.selfhostedcloud.org/shc-apps/install/latest/install.sh |
| the apps index | …/packages/generic/shc-apps-index/0.0.0-latest/index.yaml | https://downloads.selfhostedcloud.org/shc-apps/apps/latest/index.yaml |
… is https://git.corp.bitspur.com/api/v4/projects/22. The knobs:
scripts/install.shreads only a static tree:RELEASE, then the package and checksum URLs it constructs underSHC_INSTALL_BASE. Unset, that is the new line’shttps://downloads.selfhostedcloud.org/shc; set tohttps://downloads.selfhostedcloud.org/shc-binariesit installs an old-line release. Same denied-release list either way.- terraform’s
bootstrap.package.deb_url(and every example’sdeb_urldefault) is already a plain URL; the mirror’slatest/line above is the drop-in alternate. shc repo add shc <index url>takes the mirror’sapps/latest/index.yaml; itsarchive:lines already point at the mirror.
Flip a consumer only after GET https://downloads.selfhostedcloud.org/shc-binaries/latest/shc_latest_COMMIT_SHA
answers 200: the plugin reaches the estate with the SHC-707 daemon roll, and
until then the Pulp copies exist at
https://artifacts.bitspur.com/pulp/content/shc-binaries/ and …/shc-apps/.
Taking Pulp private: artifacts.corp.bitspur.com (SHC-862)
Section titled “Taking Pulp private: artifacts.corp.bitspur.com (SHC-862)”Once the mirrors above carry every public artifact, Pulp itself withdraws
behind the VPN. On the bitspur estate the corp name is already the
deployment’s first host (access: open, with pulp guarding every surface
itself) and artifacts.bitspur.com is the second, public one, so the move
is a withdrawal, not a rename. It is the LAST step of a strict order:
- the daemon roll that makes
pulp_mirrorlive on the estate (SHC-707); - bind the targets (
-C mirror.<bucket>=cloudflare) and their rules, let the buckets fill, and proveapt-get updateagainstapt.basedlinux.comwithpulp-contentstopped; - flip the consumers to the mirrors:
BASED_APT_HOSTandBASED_DOWNLOADS_HOSTon Based CI,SHC_INSTALL_BASEfor the installer, the omarchy update-channel script, the docs; - only then, the move below.
Two facts decide what “private” buys. With s3.directDownloads on, every
/pulp/content GET answers 302 to a presigned URL on the artifact store, so
a private Pulp restricts who can obtain that redirect and nothing about the
store: clients behind the VPN still need egress to it, and the store’s own
exposure is its own question. And exposure: lan enforces nothing while
lanEnforcement is off (bitspur runs it off today): the allowlist has to
be armed, and the runners that publish into Pulp have to be on it.
The move, in order
Section titled “The move, in order”# 1. arm the LAN allowlist and admit the two CI runners' source addresses# (build01's egress and the Mac runner's), or their VPN addresses.# -t is the root tenant flag, for this command only.shc -t corp config set lanAllowlistCidrs '<runner-1>/32,<runner-2>/32' --scope tenantshc -t corp config set lanEnforcement allowlist --scope tenant# 2. the content origin every distribution's base_url followsshc -t corp update pulp --var contentOrigin=https://artifacts.corp.bitspur.com# 3. the route set becomes the corp name alone. A bare '!' clears every route# and every published port, so a port this command does not restate is# removed. The public host's managed# DNS record goes with the route, and so does its ungated npm metadata# router, which ce renders per routed host (app.yaml names none)shc -t corp update pulp --expose '!' --expose artifacts.corp.bitspur.com=pulp-web:8080 --private# 4. producers follow one variable each: PULP_URL=https://artifacts.corp.bitspur.com# on the based, shc, packages and cody CI projects; BASED_APT_HOST=# https://artifacts.corp.bitspur.com/pulp/content on the Based image builds# (os/mirrors.yaml follows it); PULP_API for pulpviewVerify: an off-estate GET https://artifacts.corp.bitspur.com/pulp/content/
answers 403; the same GET from each runner answers 302 to the store; one
publish job on each runner is green. Rollback is the reverse and each step
is its own: shc -t corp config set lanEnforcement off --scope tenant opens
the router on the next 60 s sweep, --expose artifacts.bitspur.com=pulp-web:8080
re-adds the public route and its record, --var contentOrigin= returns the
default, and the CI variables go back.
What stays on Pulp after the move, and therefore behind the VPN: the dev channels, the rpm channels, the OBS build-time mirror, the npm metadata route, and the API itself.