Skip to content
SHC Docs

Object storage for apps

The catalog has no S3 server of its own. An app that needs object storage takes it from a hosted provider: Backblaze B2, Hetzner Object Storage, Cloudflare R2, AWS S3, or any other S3-compatible endpoint you rent or run somewhere else.

Three pieces sit between the provider and the app:

  1. A connection of type s3 holds one key pair and the endpoint. SHC keeps it in the vault and never writes it into a file an app can read.
  2. The external-s3 app is a headless front bound to that connection. It serves the s3 socket and runs nothing.
  3. The app integrates with the front the way it would with any provider stack: -i s3=<front>.

The bucket is yours to make. external-s3 adopts a bucket that already exists. It never creates one, never empties one and never deletes one.

An app can also bind straight to the connection with -i s3=@<connection> and no front. That is shorter, and it skips the front’s bucket check: see Straight to the connection.

At the provider, create the bucket and a key that can read and write it.

ProviderEndpoint hostRegion
Backblaze B2s3.<region>.backblazeb2.comthe bucket’s region, for example us-west-004
Hetzner Object Storage<location>.your-objectstorage.comthe location: fsn1, nbg1 or hel1
Cloudflare R2<account-id>.r2.cloudflarestorage.comauto
AWS S3s3.<region>.amazonaws.comthe bucket’s region

Two things to get right here:

  • The bucket name is global at most providers. nextcloud is taken. Use a name nobody else has, such as the app’s hostname with dashes (cloud-example-com).
  • Scope the key at the provider when you can. On Backblaze B2, a grant whose boundary names buckets mints a per-app application key restricted to those buckets. The connection key must be allowed to writeKeys, deleteKeys, listKeys and listBuckets, and to the object capabilities that per-app key is given (listBuckets, listFiles, readFiles, writeFiles, deleteFiles): B2 will not let a key create a stronger one. An unscoped grant still shares the connection’s own key with every app on it. Hetzner Object Storage has no per-bucket key API: a key opens every bucket in its project by default, and only a bucket policy at Hetzner narrows it.
Terminal window
$ shc connection connect b2 --type s3 \
--cred s3_endpoint=s3.us-west-004.backblazeb2.com \
--cred region=us-west-004 \
--value-file b2.json

s3_endpoint is the bare host: no https://, no port, no path. Add --cred port=<n> and --cred use_ssl=false only for an endpoint that is not TLS on 443.

b2.json holds the key pair, {"access_key": "<keyID>", "secret_key": "<applicationKey>"}. A secret never goes on the command line, where the shell history and the process list keep it: --cred secret_key=<applicationKey> is refused with X400548CRDSEC. Only the coordinates (s3_endpoint, region, port, use_ssl, probe_bucket) are typed inline. The secret comes from the file --value-file names, the same flag shc vault set takes. One secret is --cred secret_key=- --value-file b2.key. Several fields are one JSON object in the file named by --value-file alone. Without --value-file the same two forms read standard input: --cred secret_key=- at a terminal prompts with echo off, and --cred - takes the JSON object from a pipe. One file or stdin per call, and a field may come from one place only.

SHC checks the connection before it stores it. It lists buckets with the key. A dead endpoint, an unknown key or a wrong secret refuses the connect with X400461HLDRCH. A key scoped to one bucket may be denied the bucket list, and Cloudflare R2 answers that denial the same way whether the secret is right or wrong, so a denial proves nothing. Name a bucket the key can list with --cred probe_bucket=<bucket> and SHC lists that bucket instead; a denied key with no probe_bucket is refused, and the refusal names the field. A probe_bucket you give is always listed, so a mistyped bucket is refused at connect, not at the first app.

A connection serves nobody until it is granted. Name the tenant whose apps may use it:

Terminal window
$ shc connection grant b2 --capability s3 --grantee <tenant> \
--boundary '{"unscoped":true}'

{"unscoped":true} is an acknowledgement, and SHC makes you write it: the tenant’s apps receive the connection’s own key pair, not a key SHC cut down for them. A grant with no --boundary is refused with X403922CONUNS.

An unscoped key is shared. Every app bound to that grant, through any front, gets the same key and can read and write every bucket that key reaches. Two apps on one unscoped connection can read each other’s objects. To keep apps apart on a provider SHC cannot scope (Hetzner), give each its own bucket, its own key limited to that bucket at the provider, and its own connection and front. On Hetzner that means a bucket policy or a separate project.

--boundary '{"buckets":["<name>"]}' asks SHC to mint a key restricted to those buckets instead. That works where the provider has an API for it and SHC has a minter: Exoscale SOS, Backblaze B2, and R2 through a Cloudflare connection, which creates the bucket too. On B2 the connection key must be allowed to writeKeys, deleteKeys, listKeys and listBuckets. SHC deletes any existing application key of the same name and creates a fresh one, because B2 returns a key’s secret only once. The minted key can list, read, write and delete objects in the named buckets and can HeadBucket; it cannot manage keys or buckets. On any other endpoint the bind is refused with X403465HLDNOSCOPE, which names the endpoints that can be scoped. For Hetzner, limit the key with a bucket policy or a separate project, and grant unscoped.

Terminal window
$ shc install external-s3 -t <tenant> --stack b2 -C s3=b2

-C s3=b2 binds the app’s declared s3 connection to the connection named b2.

One optional variable, --var bucket=<names>, lists the bucket names this front is willing to serve, separated by spaces. An app that asks for any other name is refused with X5037S3PIN, which turns a typo in an app’s bucket variable into a refusal. It has a second effect: a listed bucket is also what the front’s backup captures. Read Backups before you set it.

This install starts no container and checks no bucket. The check comes when an app binds. If the connection is not granted to the tenant (step 2), the install is refused here with X403910CONGRT.

Terminal window
$ shc install nextcloud -t <tenant> --stack cloud -i postgres=<postgres stack> -i s3=b2 \
--var s3.bucket=<bucket> --var s3.accessKey= --var s3.secretKey= \
--expose cloud.example.com

-i s3=b2 names the front’s stack. Three variables go with it:

  • s3.bucket is the bucket you made. Every catalog app ships a default name, and at a hosted provider that default belongs to somebody else.
  • s3.accessKey and s3.secretKey are set empty. An app’s defaults name a key of its own, which a provider that creates users would have made for it. A hosted provider creates no user, so the app has to use the socket’s key pair, and it does that when its own two are empty.

The variable names differ between apps. Read the app’s vars.yaml before you type them.

Before the app’s containers start, the front’s probe runs against the provider with the connection’s key. It makes three kinds of read-only call (HeadBucket, a bucket listing when the first answer is “not found”, and ListObjectsV2 for one key), retries while the endpoint does not answer, and does this for every bucket the app asks for. It writes nothing. If it fails, the install stops before the app’s containers start. The edges named before s3 on the command line have already run their jobs by then, so a Postgres edge has made its database.

RefusalWhat it means
X5034S3NOBThe bucket does not exist. Make it at the provider. SHC will not.
X5035S3DENThe key was rejected for that bucket, or cannot list it.
X5036S3UNRThe endpoint did not answer.
X5037S3PINThe app asked for a bucket the front’s bucket list does not name.
X5038S3CFGThe endpoint carries a scheme, a port or a path, or the region or the bucket name is empty.

An app that needs several buckets, such as GitLab, needs every one of them made first. The probe checks each.

Binding moves no data. It changes where new objects go and leaves what the app already wrote where it is. For a running deployment: stop writes, copy the data into the bucket, then

Terminal window
$ shc update <stack> -i s3=b2 --var s3.bucket=<bucket> \
--var s3.accessKey= --var s3.secretKey=

Make the new key at the provider, write the new pair into b2.json, then connect again under the same name:

Terminal window
$ shc connection connect b2 --type s3 \
--cred s3_endpoint=s3.us-west-004.backblazeb2.com \
--cred region=us-west-004 \
--value-file b2.json

SHC checks the new key the same way before it stores it. Each bound app picks it up on its next shc update <stack>, whether it binds the front or the connection itself. Until then it keeps the old key, so delete the old key at the provider only after every bound app has been updated.

A key SHC minted for one app does not follow the connection’s key: s3-bucket on an aws connection, a bucket-scoped grant on Exoscale or Backblaze B2, or R2 through a Cloudflare connection. It stays valid until the app is uninstalled.

The front has no volume. What it can back up is the bucket itself: its backup copies the bucket’s objects into SHC’s backup repository.

  • As shipped it is off. With neither bucket nor backup.buckets set, a backup of the front records completed with the cause degraded and copies nothing. The bucket is the provider’s to keep safe.
  • Naming a bucket turns it on. --var bucket=<name> (the list in step 3) or --var backup.buckets=<name> both do. Then you must also say where the archive is built: --var backup.archiveFreeBytes=same when the backup directory is on the filesystem the capture downloads to (the stock layout), or the free bytes on the archive side when the two are split. Without it every backup of the front fails with X5044S3SPC.
  • A capture downloads the whole bucket. It needs free disk for about four times the bucket’s size on the node, plus a 20 GiB reserve, and it refuses before it downloads a byte when that room is not there.

shc backup create --all-connected on an app takes the front with it, so the same rules decide whether the app’s group backup is whole, degraded or refused.

Uninstalling an app, or disconnecting it with shc update <stack> -i s3!, leaves the bucket and everything in it. So does uninstalling the front. To drop the data, delete the bucket at the provider.

Remove the pieces in reverse: the apps that bind the front, the front (shc uninstall b2), the grant (shc connection revoke-grant b2 --capability s3 --grantee <tenant>), then the connection (shc connection rm b2).

external-s3 also takes the endpoint and the key as its own variables (--var endpoint=, --var region=, --var accessKey=, --var secretKey=) when you bind no connection. The connection is the better home for a key: it is stored once, probed when you connect it, and granted per tenant.

-i s3=@<connection>, with the @, binds the app’s s3 plug to the connection itself. There is no front to install. The grant from step 2 is the same.

Terminal window
$ shc install nextcloud -t <tenant> --stack cloud -i postgres=<postgres stack> -i s3=@b2 \
--var s3.bucket=<bucket> --expose cloud.example.com

The daemon serves the capability the connection’s type offers. An s3 connection serves s3. An unscoped grant serves the connection’s own key, with the app keeping its own bucket, so s3.bucket is needed as in step 4. A bucket-scoped grant on Exoscale or Backblaze B2 serves a per-app key instead, minted for the buckets the app’s plug addresses (its buckets, or else its bucket), not for every bucket the grant names. A bucket the plug addresses outside the boundary refuses the bind with X403488HLDPLUG, so two apps of one tenant on one grant never share a key. An aws connection serves s3-bucket: it makes a bucket and an IAM key for the app, and the bucket it made replaces s3.bucket. On an unscoped s3 grant the shared-key rule in step 2 holds here too.

The two empty key variables from step 4 are not needed here. On a connection edge SHC writes the key the connection serves over the app’s default s3.accessKey and s3.secretKey. A key variable you set to a non-empty value still wins. An app bound through the front gets no such help and still needs them empty (SHC-1464).

What the front gives you that this edge does not:

  • No check before the app starts. Nothing runs HeadBucket first, so a missing bucket or a rejected key shows up in the app’s own log, not as X5034S3NOB or X5035S3DEN at install.
  • No bucket list. Only the grant’s boundary limits what the app may ask for; there is no X5037S3PIN.
  • No bucket in backups. The edge has nothing to capture, and backup.coverage counts the bucket as covered: an s3 connection is durable storage.

Every update of the app asks the connection for its key again, so a key you rotate in the connection reaches the app on its next update (see Rotating the key). A key SHC minted for the app alone, s3-bucket on an aws connection or a bucket-scoped grant on Exoscale or Backblaze B2, is the exception: SHC keeps it in the tenant’s vault and reads it back, so an update never mints a second one while the app addresses the same buckets. An update that changes them mints a new key for the new set. If the connection behind the name changes owner, or now serves a different capability (an aws connection under the old name), the next update refuses with X409923CONMID. Uninstalling the app leaves the bucket, and the key stays in the connection.