Skip to content
SHC Docs

Pulp npm metadata overlay

SHC-730 adds an optional image for pulp-content. images.pulpContent defaults to empty and falls back to images.pulp. API, workers and initialization keep their existing image. Enabling the overlay does not migrate the database or publish packages.

The image requires pulpcore 3.114.2, pulp-npm 0.9.0, semantic-version 2.10.0 and Python 3.11. Its base is images.pulp by the same tag and digest, so the content service keeps the fork’s pulpcore and plugins and differs from the API and workers only by the overlay. TestPulpContentOverlayBaseIsServerImage fails a re-pin of images.pulp that leaves the Dockerfile behind. The build verifies both npm source hashes before replacing only NpmDistribution.content_handler with a delegate. Source drift fails the build. The inherited runtime command and user are retained. Image labels record the base, source hashes and revision.

Each advertised version comes from the stored gzip artifact’s canonical, regular package/package.json. Dependencies and other manifest fields survive; server-owned identity, distribution URL and digests replace packed values. Conditional exports and imports retain their packed declaration order, including nested conditions and array fallbacks, on cold and cached responses. Reordering these keys can change which target a consumer resolves. An image update does not remove old shared response-cache entries; targeted invalidation and public metadata/cold-install verification remain rollout requirements. hasInstallScript includes the implicit binding.gyp installation case, and _hasShrinkwrap records an actual regular shrinkwrap file. SHA-256 must match the Pulp Artifact; SHA-512 integrity and SHA-1 shasum are computed from those same bytes. There is no remote registry or source-publication fallback.

Both JSON and install-v1 requests receive the complete JSON document. Version times are the actual Package.pulp_created instants. time.created is the earliest represented creation, and time.modified is the selected repository version’s creation. These are Pulp ingestion times, not original npm publication dates. GET, cache misses and re-association never change a version’s time. latest is the greatest stable strict SemVer. A prerelease-only package uses the greatest prerelease. Equal precedence uses the original version string as a deterministic tie break. Arbitrary dist-tag CRUD isn’t provided.

Pinned repository versions remain pinned. Queries constrain both package domain fields and repository membership, and require one stored artifact in the same domain. Hosted metadata refuses repositories or distributions with remotes. Scoped package paths accept percent encoding. Tarball paths, including prerelease/build names, return to the existing core artifact handler. Content guards remain upstream of the delegate. URLs use the configured HTTPS content origin, domain and base path plus the actual ContentArtifact relative path.

Per archive: 64 MiB compressed, 256 MiB expanded, 20,000 members and 1 MiB per manifest or tar extension. Recorded Artifact.size and the configured storage object size must be positive integers, agree and fit the compressed limit before opening the file. This preflight precedes the S3 adapter’s eager body download. The byte reader and decompression limits still apply; no files are extracted. Invalid gzip/tar/PAX, sparse archives, ambiguous manifest aliases and identity mismatches fail closed. Only the canonical package root is supported. Per document: 1,024 versions and 16 MiB of serialized versions.

The process-local LRU stores at most 128 entries and 32 MiB of serialized artifact-derived data. Its key includes schema, domain, artifact identity and SHA-256. Failed validation isn’t cached. Returned objects are detached; repository membership, URLs and timestamps are rebuilt for every request. Cache misses are serialized per process to bound concurrent archive work. Metadata failures raise JSON aiohttp HTTP errors, which escape Pulp’s cache entry method. Ordinary error responses would be cached even with no-store. Successful responses retain core caching; invalidation after publishing and edge cache behavior still require verification during rollout.

The reviewed 34-package 7.8.0 archive fixture passed complete manifest and digest comparison. Its cached metadata occupied 88,066 bytes. Local filesystem timings are not S3 performance measurements. A newly published non-exempt package must wait for its true creation time plus the consumer’s release-age interval. Preserve minimumReleaseAge: 7200 and the existing exclusion list, including @multiplatform.one/*. mp.one is not covered by the exact multiplatform.one exclusion. Never backdate content or weaken the policy to make an install pass.

From the repository root:

Terminal window
uv run --no-project --python 3.11 --with pytest --with semantic-version==2.10.0 --with aiohttp pytest docker/pulp-npm-metadata/tests/test_metadata.py
uv run --no-project --with semantic-version==2.10.0 --with aiohttp python docker/pulp-npm-metadata/tests/cold_install_probe.py --consumer /path/to/consumer --output /tmp/pulp-cold-unique
uv run --no-project --with semantic-version==2.10.0 --with aiohttp python docker/pulp-npm-metadata/tests/verify_archives.py --train /path/to/reviewed/train.json --output /tmp/pulp-archive-proof.json
make -C docker build/pulp-npm-metadata BAKE_OUTPUT=docker DOCKER_REGISTRY=shc-local DOCKER_TAG=SHC-730
CGO_ENABLED=0 go test -C tests -count=1 ./unit/catalog -run Pulp
bats tests/scripts/images-publish.bats

The image-only test_s3_storage.py and test_core_cache.py regressions use the installed Django S3 adapter and Pulp cache methods with recording/in-memory transports. Run these alongside test_metadata.py inside the built image with pytest available and networking disabled. They require no Redis or credentials beyond the existing static synthetic S3 fixture values.

The cold harness requires pnpm 10.32.1 selected in the consumer checkout. It copies only the effective age/exclusion/resolution policy and uses separate user/global npm config paths, stores and caches. It never changes HOME. Its loopback HTTP fixture rewrites only its own synthetic HTTPS tarball origin for local transport. Controls cover missing timestamps, complete dependency/peer/ optional metadata, exports, corrupt tarballs, missing dependencies, release age and engine refusal. Scripts stay disabled. Explicit highest/time-based controls are separate from the normal-policy run.

docker/pulp-npm-metadata/tests/compose.yaml provides an isolated Postgres, Versity S3 Gateway and Redis stack with the overlay only on content. It requires PULP_TEST_ENCRYPTION_KEY pointing to an existing isolated test key, mounted read-only. Do not use an estate key. No key generator is included. All other credentials are static synthetic fixture values. Ports bind only to loopback.

Terminal window
PULP_TEST_ENCRYPTION_KEY=/path/to/existing/test.key docker compose -p shc730-test -f docker/pulp-npm-metadata/tests/compose.yaml up -d
uv run --no-project --with semantic-version==2.10.0 --with aiohttp python docker/pulp-npm-metadata/tests/integration_probe.py --output /tmp/pulp-integration.json

The integration probe refuses non-loopback targets and uploads synthetic archives through the publisher’s multipart REST endpoint. It checks actual metadata and S3 tarball delivery, pinned/latest membership and a warm-cache subsequent upload. It leaves fixture resources for inspection. This probe has not yet completed: the implementation run stopped at Pulp’s missing test encryption-key precondition. Real content-guard refusal, edge caching and cold pnpm against the patched Pulp stack remain acceptance work.

The catalog’s ungated_npm_metadata declaration binds GET/HEAD metadata reads to the pulp-web service and the exact 34 @multiplatform.one package names under /pulp/content/multiplatform.one/. It names no host: CE renders the sibling on every host the deployment routes to pulp-web, so the hosts come from the deployment’s own --expose (SHC-1769). An entry that still carries a host key is ignored. The declaration is independent of package versions. Adding another name is a catalog policy change; an arbitrary package or another distribution does not inherit the exemption.

CE reads the declaration during both install and ingress reconciliation. It emits a sibling at priority 89001 using the existing service and TLS configuration. The scope/name separator accepts / or %2F, including lowercase encoding, and the scope marker accepts @ or %40. A full-path match excludes tarballs and suffixes. The original ungated_paths traversal guard stays unchanged, including its encoded-slash refusal.

The new sibling inherits the renderer’s gate-cookie stripping and LAN access policy. It carries no forward-auth gate or response injector. Pulp still owns distribution authorization and content guards; this route neither grants a role nor turns a private distribution public. Decoded non-read requests can still reach the original content-prefix router, where Pulp enforces its method contract. No credentials are added to public discovery.

Traefik 3.6.25 normalizes unreserved escapes and dot segments before routing. Normalization-equivalent canonical public names are accepted. Encoded percent remains reserved, so double-encoded separators do not gain this exemption. Destination isolation is the contract; original wire spelling is not retained for a later router to inspect.

Validation uses the supported CE overlay runner against an isolated CE checkout:

Terminal window
SHC_CE_DIR=/path/to/reviewed-ce sh scripts/ce-test.sh -run 'TestNPMMetadata' ./modules/app
CGO_ENABLED=0 go -C tests test -mod=readonly -count=1 ./unit/catalog -run TestPulpNPMMetadataIngressDeclaration

TestNPMMetadataPinnedTraefik is opt-in and skips without CE_NPM_TRAEFIK_BINARY. Supply an independently checksum-verified upstream Traefik 3.6.25 executable to run the actual entrypoint/router and this checkout’s cookie-strip plugin against loopback content/gate fixtures:

Terminal window
SHC_CE_DIR=/path/to/reviewed-ce \
CE_NPM_TRAEFIK_BINARY=/path/to/verified-traefik-3.6.25 \
sh scripts/ce-test.sh -v -run '^TestNPMMetadataPinnedTraefik$' ./modules/app

The fixture uses the real catalog allowlist and tests all names with encoded and decoded separators, non-read methods, neighboring paths and the original OpenReplay traversal cases. Its upstream 404/403 and gate 401/302 responses are synthetic markers. They prove routing and response preservation, not actual Pulp content-guard execution or live edge behavior. The parent must confirm the deployed edge version and retain the separate Pulp guard/publication proof.

This declaration requires the CE renderer revision that understands it. Release that revision and advance SHC’s CE dependency pins before deploying the catalog; an older daemon ignores the new field and encoded metadata still receives 401. The developer CE overlay is not a release pin. Removing the declaration and reconciling removes only this sibling and restores the original route set. Do not patch generated ingress YAML or relax the shared traversal guard.

The parent owns publication and deployment after independent review. Capture the existing deployment image and repository/distribution state first. Publish through the existing images/core Bake pipeline, then set only images.pulpContent to the reviewed immutable image digest. Review the rendered service diff and run the real train’s metadata/tarball and cold-client checks. Also verify the existing apt/file supply paths, which share the content process. Invalidate only proven stale entries for the affected npm distribution, never the whole Redis cache.

The images.pulpContent override is persistent deployment state, so a later re-pin of images.pulp doesn’t reach the content service on its own. Re-point the override to the overlay image built from the re-pinning commit in the same update.

Rollback restores the captured images.pulpContent override, empty if it was previously absent, and reconverges content only. There is no data migration to reverse. Rollback also restores the original npm metadata limitation, so leave consumer cutover blocked until registry acceptance passes.