Deploy and run the hub

The hub is a small server that keeps an organisation’s projects in sync and manages who can use them. One hub serves every organisation on it. Each organisation gets its own address, for example https://sift.fathom.operationalecology.io.

This page summarises how the hub is deployed and run. The full step-by-step runbook, with every command, secret and check, is docs/DEPLOY.md in the Sift repository. Follow that when you deploy. Prices and provider screens change, so treat the details here as a guide.

What runs where

flowchart LR
  DNS["DNS<br/>wildcard record,<br/>DNS only"] --> CAD
  subgraph SERVER["One server, docker compose"]
    CAD["Caddy<br/>HTTPS for each<br/>organisation address"] --> API["Hub API<br/>and hub web app"]
    API --> PG[("Postgres<br/>people, projects,<br/>labels as ops")]
    API --> BLOB[("Blob store<br/>snapshots and<br/>version exports")]
  end
  API -- "invitations and notices" --> MAIL["Mail server<br/>(SMTP)"]
  SERVER -- "nightly encrypted backup" --> BK[("Backup bucket<br/>(Cloudflare R2)")]
  APPS["Desktop apps"] --> CAD
  WEB["Browsers"] --> CAD

The hub never receives audio. It holds people, organisations, projects, the stream of label changes from every device, and published versions. Recordings stay on people’s computers.

What you need

The reference deployment uses:

• A server: an AWS Lightsail instance with Ubuntu 24.04, 2 GB of memory and a static IP. A launch script in the repository (deploy/lightsail-launch.sh) sets up Docker, the firewall (ports 22, 80 and 443), automatic security updates and a deploy user.
• DNS on Cloudflare, with the domain still registered wherever it is.
• Two Cloudflare R2 buckets: one public, for the desktop app’s installers, and one private, for backups.
• The hub image, built by GitHub CI from a hub-v* tag and pulled from the GitHub container registry with a read-only token.
• A mail account that can send over SMTP, for invitations and password resets.
• A password manager for the secrets you generate along the way.

Budget about two hours, most of it waiting for DNS.

DNS and HTTPS

One wildcard A record, *, points at the server, with the proxy turned off (“DNS only”). It covers every organisation’s address and the hub’s own page. Never add records named after an organisation.

Caddy obtains a certificate for each address the first time someone visits it, and renews it automatically. Before it does, it asks the hub, so only the addresses of real organisations ever get a certificate.

Configuration

The settings live in /opt/hub/deploy/.env on the server, copied from .env.example. You generate three secrets (the token signing secret, the database password and the setup token) and set the image, the domain, the address pattern and the public URL. Keep a copy of the finished file in your password manager. Never commit it.

Leave HUB_REGISTRATION_OPEN=false so that nobody can create an organisation on their own.

Mail

Without mail settings the hub sends no email, and the command line prints links for you to send. To send email, set HUB_MAILER=smtp, the HUB_SMTP_* settings and HUB_MAIL_FROM. Set up SPF, DKIM and DMARC records for the sending domain so the mail arrives. Test by inviting yourself from the People page.

First start

Start the stack with docker compose up -d (the runbook uses a dc alias for the long command). The hub applies its database migrations as it starts.

With registration closed, create the first accounts with the hub’s command line on the server:

1. hub create-admin: yourself, as hub admin.
2. hub create-org: the organisation and its owner. If the owner is you, you can sign in at once. Otherwise it prints an invitation link to send to the owner.
3. hub invite: prints an invitation link for someone, when the hub has no mail set up.

After that, the owner invites everyone else from the web app. See Set up an organisation.

Short names (slugs) are 3 to 48 lowercase letters, digits or dashes. Names such as www, api, downloads, admin, mail, hub, status and sift are reserved.

Check that it works

• /health at an organisation’s address answers {"status":"ok"}.
• /ready at the hub’s address answers with "database":"ok".
• The organisation’s address shows Sign in to followed by the organisation’s name in a browser.
• In the desktop app, the hub address is the organisation’s address, for example sift.fathom.operationalecology.io.

If a certificate doesn’t appear, read Caddy’s log. The usual causes are DNS not pointing at the server, ports 80 or 443 closed in the provider’s firewall, or Let’s Encrypt rate limits after many failed attempts.

Backups and the restore drill

Every night at 03:30 UTC, a backup script dumps the database, checks the dump, takes a base backup, and stores them with the blobs in an encrypted restic repository in the private R2 bucket. It keeps 7 daily, 4 weekly and 6 monthly backups, and on Sundays it verifies a sample of the stored data.

• Put the backup password in your password manager the moment you create it. Without it, nobody can read the backups.
• The hub’s own signing key (HUB_SIGNING_KEY in .env) is not in the backups on purpose. Keep it in your password manager too. After a restore, the apps send back changes the restored hub lost, with the hub’s signed receipts for them; the hub checks those receipts with this key. To change the key, set a new one and list the old public key in HUB_SIGNING_KEYS_RETIRED. If the old key was stolen or leaked, don’t list it: after a restore, changes it vouched for come back only from their authors’ own computers.
• Optionally, add a free dead-man’s switch (healthchecks.io) that emails you when a night’s backup fails or doesn’t run.
• Run the restore drill now, and then every month: restore.sh drill. It restores the latest backup into a throwaway database beside production, counts rows, checks every blob is there, and ends with drill passed. It touches nothing in production.

The database also archives its write-ahead log continuously, and a second timer ships it to the backup bucket every 5 minutes, so a restore can go back to any minute of the last week.

Restoring for real: restore.sh pitr 'YYYY-MM-DD HH:MM:SS' brings the database back to that minute (UTC), from the night’s base backup and the archived log. It recovers into a new volume beside the live database and swaps it in only once recovery worked, keeping the old data in a volume you remove afterwards. restore.sh production puts back last night’s backup. Both stop the hub, restore, and re-mint the hub’s sync epoch before the apps can reconnect. Each app then fetches everything again and sends back any of its own changes the restored hub no longer has, and others’ changes it holds under their authors’ signatures, so work that reached the hub after the restore point is not lost as long as a computer that holds it syncs again. A dataset or project the restored hub lost is added again by a manager’s or an organisation admin’s app; its members and access must then be set again in the hub web app. To rebuild a lost server, create a new one, move the static IP to it, restore .env and the backup settings from your password manager, start the empty hub, then restore.

People’s desktop apps hold copies of the projects they work on, but those copies are not a backup.

Hub settings in the hub web app shows whether backups are reporting: the time of the last good backup, or a warning when one failed, none has finished, or the last is more than two days old. Oldest restore point shows the time of the oldest retained backup, the earliest point a restore can reach.

Device pruning

Device pruning is off by default. Leave it off unless disk space on people’s computers becomes a problem. With it off, every device keeps the full history of the datasets it holds: a disaster copy beside your backups.

A hub admin turns it on or off for the whole hub, every organisation at once, in the hub web app: Hub settings ▸ Device pruning ▸ Turn on (or Turn off). The server setting HUB_DEVICE_PRUNING sets the starting value; the switch overrides it either way, and restoring the database doesn’t change it. With it on, devices drop the data of old changes, but only changes that the oldest retained backup holds. The setting reads “On: devices drop old changes every retained backup holds”, or “On, waiting for a backup that records what it holds” until a nightly backup has recorded that.

While pruning is on:

• Restore only with the restore script, to a point no older than the oldest retained backup (Oldest restore point in Hub settings). Every such restore holds every change a device dropped.
• Turn it off before a restore from anywhere else (a backup retention already removed, a copy from outside the backup repository, a hand-made dump) is ever needed, for example while you plan a move to another server. Turning it off stops further dropping; what a device already dropped stays dropped.
• If such a restore happens anyway, each device that dropped history shows “The hub was restored from an older backup” on the affected dataset. Nothing is lost on that device. The person exports “the work only this computer holds”, a manager imports the file, and then they start the dataset over from the hub. Where the hub lost something an import can’t bring back (the dataset itself, a detector run, a source or its recordings, a version), there is no start over: the file is a full backup of the dataset, and it comes to you. Such a file can’t be loaded into a hub yet, so keep it as an archive, ask that person not to start over or reinstall, and copy their Sift data safely (the runbook says which files, copied while Sift is closed). See Restored from an older backup.

Monitoring

Use a free external uptime monitor on two addresses: the hub’s /ready (looking for "database":"ok") and one organisation’s /health, which also fails if that certificate stops renewing. Together with the backup check, you hear when the hub is down, its database is down, or backups stop.

Updates

• The hub: tag a new hub release, wait for CI to build the image, then run ./update.sh <version> on the server. It backs up first, pulls the image, applies the database migrations, restarts the hub and waits until it is ready. To roll back, run it with the older version. If the newer version changed the database, restore the pre-update backup first. Anything written since the update is lost then.
• Caddy and Postgres: pull new images monthly for security fixes. A Postgres major upgrade is a dump and restore, not a pull.
• Ubuntu installs security updates by itself and reboots at night when it needs to.

The desktop app has its own releases. See Releases and updates.

Costs

The reference deployment costs about $12 to $14 a month: $12 for the Lightsail instance, and a dollar or two for optional instance snapshots. Cloudflare DNS, R2 at this size and the container registry cost nothing at first. Check current prices before you commit.

Try it on a laptop first

deploy/test-local.sh runs the whole stack on your computer with a fake domain and Caddy’s own certificate authority. It checks HTTPS, live updates, a backup, a restore drill, a restore over production and an update, then removes everything again.