Object storage for apps
The catalog has no S3 server of its own. An app that needs object storage takes it from a hosted provider: Backblaze B2, Hetzner Object Storage, Cloudflare R2, AWS S3, or any other S3-compatible endpoint you rent or run somewhere else.
Three pieces sit between the provider and the app:
- A connection of type
s3holds one key pair and the endpoint. SHC keeps it in the vault and never writes it into a file an app can read. - The
external-s3app is a headless front bound to that connection. It serves thes3socket and runs nothing. - The app integrates with the front the way it would with any provider
stack:
-i s3=<front>.
The bucket is yours to make. external-s3 adopts a bucket that already
exists. It never creates one, never empties one and never deletes one.
An app can also bind straight to the connection with -i s3=@<connection>
and no front. That is shorter, and it skips the front’s bucket check: see
Straight to the connection.
Before you start
Section titled “Before you start”At the provider, create the bucket and a key that can read and write it.
| Provider | Endpoint host | Region |
|---|---|---|
| Backblaze B2 | s3.<region>.backblazeb2.com | the bucket’s region, for example us-west-004 |
| Hetzner Object Storage | <location>.your-objectstorage.com | the location: fsn1, nbg1 or hel1 |
| Cloudflare R2 | <account-id>.r2.cloudflarestorage.com | auto |
| AWS S3 | s3.<region>.amazonaws.com | the bucket’s region |
Two things to get right here:
- The bucket name is global at most providers.
nextcloudis taken. Use a name nobody else has, such as the app’s hostname with dashes (cloud-example-com). - Scope the key at the provider when you can. On Backblaze B2, a grant
whose boundary names buckets mints a per-app application key restricted to
those buckets. The connection key must be allowed to
writeKeys,deleteKeys,listKeysandlistBuckets, and to the object capabilities that per-app key is given (listBuckets,listFiles,readFiles,writeFiles,deleteFiles): B2 will not let a key create a stronger one. An unscoped grant still shares the connection’s own key with every app on it. Hetzner Object Storage has no per-bucket key API: a key opens every bucket in its project by default, and only a bucket policy at Hetzner narrows it.
1. Connect the provider
Section titled “1. Connect the provider”$ shc connection connect b2 --type s3 \ --cred s3_endpoint=s3.us-west-004.backblazeb2.com \ --cred region=us-west-004 \ --value-file b2.jsons3_endpoint is the bare host: no https://, no port, no path. Add --cred port=<n> and --cred use_ssl=false only for an endpoint that is not TLS on
443.
b2.json holds the key pair, {"access_key": "<keyID>", "secret_key": "<applicationKey>"}.
A secret never goes on the command line, where the shell history and the
process list keep it: --cred secret_key=<applicationKey> is refused with
X400548CRDSEC. Only the coordinates (s3_endpoint, region, port,
use_ssl, probe_bucket) are typed inline. The secret comes from the file
--value-file names, the same flag shc vault set takes. One secret is
--cred secret_key=- --value-file b2.key. Several fields are one JSON object
in the file named by --value-file alone. Without --value-file the same two
forms read standard input: --cred secret_key=- at a terminal prompts with
echo off, and --cred - takes the JSON object from a pipe. One file or stdin
per call, and a field may come from one place only.
SHC checks the connection before it stores it. It lists buckets with the key.
A dead endpoint, an unknown key or a wrong secret refuses the connect with
X400461HLDRCH. A key scoped to one bucket may be denied the bucket list, and
Cloudflare R2 answers that denial the same way whether the secret is right or
wrong, so a denial proves nothing. Name a bucket the key can list with --cred probe_bucket=<bucket> and SHC lists that bucket instead; a denied key with no
probe_bucket is refused, and the refusal names the field. A probe_bucket
you give is always listed, so a mistyped bucket is refused at connect, not at
the first app.
2. Grant it to the tenant
Section titled “2. Grant it to the tenant”A connection serves nobody until it is granted. Name the tenant whose apps may use it:
$ shc connection grant b2 --capability s3 --grantee <tenant> \ --boundary '{"unscoped":true}'{"unscoped":true} is an acknowledgement, and SHC makes you write it: the
tenant’s apps receive the connection’s own key pair, not a key SHC cut down
for them. A grant with no --boundary is refused with X403922CONUNS.
An unscoped key is shared. Every app bound to that grant, through any front, gets the same key and can read and write every bucket that key reaches. Two apps on one unscoped connection can read each other’s objects. To keep apps apart on a provider SHC cannot scope (Hetzner), give each its own bucket, its own key limited to that bucket at the provider, and its own connection and front. On Hetzner that means a bucket policy or a separate project.
--boundary '{"buckets":["<name>"]}' asks SHC to mint a key restricted to
those buckets instead. That works where the provider has an API for it and
SHC has a minter: Exoscale SOS, Backblaze B2, and R2 through a
Cloudflare connection, which creates the bucket
too. On B2 the connection key must be allowed to writeKeys, deleteKeys,
listKeys and listBuckets. SHC deletes any existing application key of
the same name and creates a fresh one, because B2 returns a key’s secret
only once. The minted key can list, read, write and delete objects in the
named buckets and can HeadBucket; it cannot manage keys or buckets. On
any other endpoint the bind is refused with X403465HLDNOSCOPE, which
names the endpoints that can be scoped. For Hetzner, limit the key with a
bucket policy or a separate project, and grant unscoped.
3. Install the front
Section titled “3. Install the front”$ shc install external-s3 -t <tenant> --stack b2 -C s3=b2-C s3=b2 binds the app’s declared s3 connection to the connection named
b2.
One optional variable, --var bucket=<names>, lists the bucket names this
front is willing to serve, separated by spaces. An app that asks for any
other name is refused with X5037S3PIN, which turns a typo in an app’s
bucket variable into a refusal. It has a second effect: a listed bucket is
also what the front’s backup captures. Read Backups before you
set it.
This install starts no container and checks no bucket. The check comes when an
app binds. If the connection is not granted to the tenant (step 2), the install
is refused here with X403910CONGRT.
4. Bind the app
Section titled “4. Bind the app”$ shc install nextcloud -t <tenant> --stack cloud -i postgres=<postgres stack> -i s3=b2 \ --var s3.bucket=<bucket> --var s3.accessKey= --var s3.secretKey= \ --expose cloud.example.com-i s3=b2 names the front’s stack. Three variables go with it:
s3.bucketis the bucket you made. Every catalog app ships a default name, and at a hosted provider that default belongs to somebody else.s3.accessKeyands3.secretKeyare set empty. An app’s defaults name a key of its own, which a provider that creates users would have made for it. A hosted provider creates no user, so the app has to use the socket’s key pair, and it does that when its own two are empty.
The variable names differ between apps. Read the app’s vars.yaml before you
type them.
Before the app’s containers start, the front’s probe runs against the
provider with the connection’s key. It makes three kinds of read-only call
(HeadBucket, a bucket listing when the first answer is “not found”, and
ListObjectsV2 for one key), retries while the endpoint does not answer,
and does this for every bucket the app asks for. It writes nothing. If it
fails, the install stops before the app’s containers start. The edges named
before s3 on the command line have already run their jobs by then, so a
Postgres edge has made its database.
| Refusal | What it means |
|---|---|
X5034S3NOB | The bucket does not exist. Make it at the provider. SHC will not. |
X5035S3DEN | The key was rejected for that bucket, or cannot list it. |
X5036S3UNR | The endpoint did not answer. |
X5037S3PIN | The app asked for a bucket the front’s bucket list does not name. |
X5038S3CFG | The endpoint carries a scheme, a port or a path, or the region or the bucket name is empty. |
An app that needs several buckets, such as GitLab, needs every one of them made first. The probe checks each.
An app that is already running
Section titled “An app that is already running”Binding moves no data. It changes where new objects go and leaves what the app already wrote where it is. For a running deployment: stop writes, copy the data into the bucket, then
$ shc update <stack> -i s3=b2 --var s3.bucket=<bucket> \ --var s3.accessKey= --var s3.secretKey=Rotating the key
Section titled “Rotating the key”Make the new key at the provider, write the new pair into b2.json,
then connect again under the same name:
$ shc connection connect b2 --type s3 \ --cred s3_endpoint=s3.us-west-004.backblazeb2.com \ --cred region=us-west-004 \ --value-file b2.jsonSHC checks the new key the same way before it stores it. Each bound app
picks it up on its next shc update <stack>, whether it binds the front or
the connection itself. Until then it keeps the old key, so delete the old
key at the provider only after every bound app has been updated.
A key SHC minted for one app does not follow the connection’s key:
s3-bucket on an aws connection, a bucket-scoped grant on Exoscale or
Backblaze B2, or R2 through a Cloudflare connection. It stays valid until
the app is uninstalled.
Backups
Section titled “Backups”The front has no volume. What it can back up is the bucket itself: its backup copies the bucket’s objects into SHC’s backup repository.
- As shipped it is off. With neither
bucketnorbackup.bucketsset, a backup of the front recordscompletedwith the causedegradedand copies nothing. The bucket is the provider’s to keep safe. - Naming a bucket turns it on.
--var bucket=<name>(the list in step 3) or--var backup.buckets=<name>both do. Then you must also say where the archive is built:--var backup.archiveFreeBytes=samewhen the backup directory is on the filesystem the capture downloads to (the stock layout), or the free bytes on the archive side when the two are split. Without it every backup of the front fails withX5044S3SPC. - A capture downloads the whole bucket. It needs free disk for about four times the bucket’s size on the node, plus a 20 GiB reserve, and it refuses before it downloads a byte when that room is not there.
shc backup create --all-connected on an app takes the front with it, so
the same rules decide whether the app’s group backup is whole, degraded or
refused.
Taking it apart
Section titled “Taking it apart”Uninstalling an app, or disconnecting it with shc update <stack> -i s3!,
leaves the bucket and everything in it. So does uninstalling the front. To
drop the data, delete the bucket at the provider.
Remove the pieces in reverse: the apps that bind the front, the front
(shc uninstall b2), the grant (shc connection revoke-grant b2 --capability s3 --grantee <tenant>), then the connection (shc connection rm b2).
Without a connection
Section titled “Without a connection”external-s3 also takes the endpoint and the key as its own variables
(--var endpoint=, --var region=, --var accessKey=, --var secretKey=)
when you bind no connection. The connection is the better home for a key: it
is stored once, probed when you connect it, and granted per tenant.
Straight to the connection
Section titled “Straight to the connection”-i s3=@<connection>, with the @, binds the app’s s3 plug to the
connection itself. There is no front to install. The grant from step 2 is
the same.
$ shc install nextcloud -t <tenant> --stack cloud -i postgres=<postgres stack> -i s3=@b2 \ --var s3.bucket=<bucket> --expose cloud.example.comThe daemon serves the capability the connection’s type offers. An s3
connection serves s3. An unscoped grant serves the connection’s own key,
with the app keeping its own bucket, so s3.bucket is needed as in step 4.
A bucket-scoped grant on Exoscale or Backblaze B2 serves a per-app key
instead, minted for the buckets the app’s plug addresses (its buckets, or
else its bucket), not for every bucket the grant names. A bucket the plug
addresses outside the boundary refuses the bind with X403488HLDPLUG, so
two apps of one tenant on one grant never share a key. An aws connection serves s3-bucket: it makes a bucket and an
IAM key for the app, and the bucket it made replaces s3.bucket. On an
unscoped s3 grant the shared-key rule in step 2 holds here too.
The two empty key variables from step 4 are not needed here. On a
connection edge SHC writes the key the connection serves over the app’s
default s3.accessKey and s3.secretKey. A key variable you set to a
non-empty value still wins. An app bound through the front gets no such
help and still needs them empty (SHC-1464).
What the front gives you that this edge does not:
- No check before the app starts. Nothing runs
HeadBucketfirst, so a missing bucket or a rejected key shows up in the app’s own log, not asX5034S3NOBorX5035S3DENat install. - No bucket list. Only the grant’s boundary limits what the app may ask
for; there is no
X5037S3PIN. - No bucket in backups. The edge has nothing to capture, and
backup.coveragecounts the bucket as covered: an s3 connection is durable storage.
Every update of the app asks the connection for its key again, so a key
you rotate in the connection reaches the app on its next update (see
Rotating the key). A key SHC minted for the app alone,
s3-bucket on an aws connection or a bucket-scoped grant on Exoscale or
Backblaze B2, is the exception: SHC keeps it in the tenant’s vault and
reads it back, so an update never mints a second one while the app
addresses the same buckets. An update that changes them mints a new key
for the new set. If the connection behind the name changes
owner, or now serves a different capability (an aws connection under the
old name), the next update refuses with X409923CONMID. Uninstalling the
app leaves the bucket, and the key stays in the connection.