Deploy orrery to prod VPS — operator runbook

GH #260 phase 1 (tailnet-only) + #261 thin-deploy pivot. Standing orrery up on the existing always-on Hetzner VPS that also hosts podcast_scraper.

Phase 2 (public DNS + TLS) is deferred — see #260 §"Phase 2".

The deploy workflow follows the same ownership split as podcast_scraper: cloud-init owns bootstrap (runs as root, once, at VPS first-boot), the workflow owns deploy (runs as deploy, idempotent, every push).

Architecture

                                     ┌──────────────────────────────────────┐
                                     │ Hetzner VPS (existing, podcast_scraper-host) │
                                     │                                      │
                                     │  /srv/podcast-scraper/  (existing)   │
                                     │     compose ↔ host :8080             │
                                     │     tailscale serve :443             │
                                     │                                      │
                                     │  /srv/orrery/        (new — #260)    │
                                     │     compose ↔ host :8090             │
                                     │     tailscale serve :8443            │
                                     │                                      │
                                     └──────────────────────────────────────┘
                                                      │
                                                      │ tailnet (no public DNS)
                                                      ▼
                              https://<host>.<tailnet>:443    ← podcast_scraper viewer
                              https://<host>.<tailnet>:8443   ← orrery (new)

Co-tenancy is enforced by four mechanics:

Mechanic	podcast_scraper	orrery
Compose project name	`podcast_scraper`	`orrery`
Host port	`:8080`	`:8090` (env: `ORRERY_PORT`)
Filesystem root	`/srv/podcast-scraper/`	`/srv/orrery/`
Tailscale serve	`--https=443`	`--https=8443`

VPS prerequisites — owned by podcast_scraper cloud-init

orrery rides on the existing tailnet node where podcast_scraper runs. The following are owned by podcast_scraper's Terraform / cloud-init in its own repo. Some pre-existed (✅); two were added for orrery (➕, tracked under chipi/podcast_scraper#834 + #261-Move-1):

✅ VPS provisioned (Hetzner CAX) with tailnet identity registered
✅ deploy@ user exists, in the docker group
✅ Docker engine + docker compose plugin installed
✅ Hetzner firewall: 22/SSH + ICMP only; all other ports private
➕ deploy@ ~/.ssh/authorized_keys includes the orrery deploy pubkey matching PROD_SSH_PRIVATE_KEY (in addition to the pre-existing podcast_scraper deploy key)
➕ /srv/orrery/ exists as a deploy:deploy-owned git checkout of chipi/orrery main

The reason the last two ride in podcast_scraper's cloud-init: terraform owns the VPS state. An imperative mkdir + cat >> authorized_keys from this repo would be overwritten on the next tofu apply. Doing it via the cloud-init that already templates the host keeps it durable across VPS rebuilds.

orrery adds zero ad-hoc steps at deploy time. The workflow assumes the checkout exists and just git fetch+reset --hard origin/mains.

The deploy flow

After the prereqs above land, every deploy is just:

SSHes into VPS as deploy@
Builds the static bundle on the runner with PUBLIC_SENTRY_* env exported (vite/SvelteKit bakes them in)
Rsyncs the CI-built build/ into /srv/orrery/build/ (the static-bundle deploy artefact — ADR-063)
Stages /srv/orrery/.env from GH prod env secrets (atomic: mktemp -p /dev/shm + trap shred + printf + scp .env.deploy-staged + atomic mv). Single source of truth — missing secrets → empty values → integrations no-op
git fetch --depth=50 origin main + git reset --hard origin/main — brings the on-VPS checkout in lockstep with the deploying commit
bash ops/deploy_stack.sh (from the fresh checkout) — HUPs the shared node Alloy (ADR-121), builds web, bootstraps lab-api-state (dir + allowlist seed + docker-mediated chown to the labapi uid), and brings up web always + mcp/lab-api only when their auth secrets are staged (else web-only + a ::warning:: — lab-api crash-loops without LAB_GOOGLE_CLIENT_ID/SECRET + LAB_CLAUDE_CLIENT_SECRET). pipeline-runner stays profiles: manual.
Per-service loopback healthchecks: web on :ORRERY_PORT, lab-api on :8094/jwks (host bind relocated from 8093 — a co-tenant operator-viewer-1 holds 8093; container-internal port stays 8093), mcp on :8091/health (when the full stack is up)
Re-arms tailscale serve via the root-owned wrapper sudo -n /usr/local/sbin/orrery-tailscale-serve.sh (narrow NOPASSWD entry; podcast_scraper#838)
External tailnet probe at https://<PROD_TAILNET_FQDN>:8443/

No sudo for the bulk of the work. No .env scaffolding by the operator. No mkdir. All operations run as the deploy@ user except the one narrow sudo -n call to the root-owned tailscale-serve wrapper.

The staged /srv/orrery/.env is RAM-rendered + shipped atomically + chmod 600 — never persists on the runner's disk. It contains static config (COMPOSE_PROJECT_NAME, ORRERY_PORT, ORRERY_PIPELINE_IMAGE_TAG). Regenerated every deploy from GH Secrets — those are the source of truth. (Log shipping has no GH secrets — handled by shared node Alloy, ADR-121.)

Caveat — tailscale-serve durability

tailscale serve --bg registers the :8443 → 127.0.0.1:8090 mapping in tailscaled's runtime state. This does not persist across tailscaled restarts (VPS reboot, tailscale package upgrade). The deploy workflow re-registers on every run, so a successful deploy after a restart re-arms the serve config.

For true persistence (survive between deploys when there is no operator action), the systemd ExecStartPost pattern from podcast_scraper should be replicated in the same Terraform / cloud-init that provisioned the VPS (which lives in the podcast_scraper repo). The snippet below is ready to paste:

yaml

# Add to podcast_scraper's infra/cloud-init/prod.user-data, alongside
# the existing podcast-tailscale-serve.sh write_files entry:
write_files:
  - path: /usr/local/sbin/orrery-tailscale-serve.sh
    permissions: '0755'
    content: |
      #!/bin/sh
      # Mirrors podcast-tailscale-serve.sh, publishing orrery's host
      # ORRERY_PORT (default 8090) on the tailnet at HTTPS :8443.
      # Source of truth: orrery repo infra/cloud-init/orrery-tailscale-serve.sh
      set -eu
      PORT=8090
      if [ -f /srv/orrery/.env ]; then
        line=$(grep -E '^ORRERY_PORT=' /srv/orrery/.env | tail -1 || true)
        if [ -n "$line" ]; then
          v=$(echo "$line" | cut -d= -f2- | tr -d ' \t"' | tr -d "'")
          if [ -n "$v" ]; then PORT="$v"; fi
        fi
      fi
      i=1
      while [ "$i" -le 60 ]; do
        if curl -fsS "http://127.0.0.1:${PORT}/" >/dev/null 2>&1; then
          break
        fi
        sleep 2
        i=$((i + 1))
      done
      /usr/bin/tailscale serve --bg --https=8443 "$PORT"
  - path: /etc/systemd/system/tailscaled.service.d/orrery-serve.conf
    permissions: '0644'
    content: |
      [Service]
      ExecStartPost=/usr/local/sbin/orrery-tailscale-serve.sh

# In the runcmd section of the same cloud-init, add:
runcmd:
  - systemctl daemon-reload
  - systemctl restart tailscaled.service

Once that's in podcast_scraper's Terraform + a tofu apply lands, the tailscale-serve registration persists across reboots automatically. Until then, every orrery deploy re-arms it — which is fine for the test-and-iterate phase.

GitHub Actions setup (one-time)

In the orrery repo settings → Environments → prod → Environment secrets/variables:

Required (deploy will skip with warning if missing)

Type	Name	Value
Secret	`TS_AUTHKEY`	Tailscale auth key with `tag:gha-deployer`. Reusable, 90-day expiry recommended.
Secret	`PROD_SSH_PRIVATE_KEY`	Ed25519 PEM (private key matching `~/.ssh/authorized_keys` of `deploy@` on the VPS).
Variable	`PROD_TAILNET_FQDN`	The MagicDNS host of the VPS, e.g. `prod-podcast.<tailnet>.ts.net`. Same host that serves podcast_scraper; orrery uses a different tailscale-serve port (:8443).
Variable	`PROD_SSH_HOST_KEY`	The VPS's ed25519 SSH host public key, `ssh-ed25519 AAAA…` (no hostname). Deploy and vps-ops pin it and connect with `StrictHostKeyChecking=yes`; the job fails if it is unset. See §"Re-pin the SSH host key".

Optional — observability (#263)

All four can be left unset; integrations silently no-op (${VAR:-} defaults in compose) and the stack still comes up. Set when you're ready to wire telemetry. Per ADR-067 / ADR-068.

Type	Name	Value
Secret	`PUBLIC_SENTRY_DSN`	GlitchTip project DSN (self-hosted ingest at `telemetry.orrerylearn.com`). Public-by-design (identifies the project, doesn't authenticate). Baked into the static bundle at build time. Note: name omits `PROD_` prefix because the workflow passes it through unchanged as a `PUBLIC_*` SvelteKit env.

Log shipping needs no GH secrets — handled by the shared node Alloy (infra repo ADR-121); see docs/guides/observability.md.

The workflow's Stage /srv/orrery/.env step renders all the runtime env (project name, port, image tag, Grafana creds) into /srv/orrery/.env atomically each deploy — mirrors podcast_scraper deploy.yml lines 147-218 exactly. Sentry vars are passed to the build step directly (PUBLIC_ prefix means SvelteKit bakes them into the bundle).

Log shipping uses the shared node Alloy (infra repo ADR-121, ops/observability/orrery.alloy) → self-hosted VictoriaLogs on the homelab Mac mini. Query orrery logs with LogsQL app:orrery in Grafana at http://homelab:3000. No per-app Grafana Cloud account needed.

Triggering a deploy

Until the workflow is upgraded to push: branches: [main], all deploys are manual:

bash

gh workflow run "Deploy to prod VPS (orrery)" --ref main

Optional input: override_image_sha to pin a specific :sha-<7> tag (useful for rollback to a known-good build).

The workflow:

Pre-flight checks the three required secrets/vars are set; warns + skips on missing.
Checks out the repo.
Resolves the target image SHA (defaults to current main HEAD short).
Builds the static bundle on the runner (npm ci && npm run build) with PUBLIC_SENTRY_DSN/PUBLIC_SENTRY_ENVIRONMENT/PUBLIC_SENTRY_RELEASE exported so vite bakes them in.
Joins the tailnet.
Installs the SSH key (validates it's a loadable OpenSSH PEM before continuing).
Rsyncs build/ to /srv/orrery/build/.
Stages /srv/orrery/.env from GH prod-scoped secrets — atomic, shm-backed tmpfile, never on the runner's disk.
SSH'es in: git fetch + reset --hard origin/main + source ./.env + docker compose --env-file .env pull/up web + loopback healthcheck on :ORRERY_PORT. (Log shipping: shared node Alloy HUP, ADR-121 — no grafana-agent step.)
Re-arms tailscale serve via the root-owned wrapper (sudo -n /usr/local/sbin/orrery-tailscale-serve.sh, 3-attempt retry, WARN-not-fail).
External healthcheck over tailnet at https://<PROD_TAILNET_FQDN>:8443/.

Total: ~3–5 min for a no-op deploy, ~5–7 min when the pipeline-runner image is a fresh SHA (new layers to pull).

Verifying

From any tailnet member:

bash

curl -fsS https://<PROD_TAILNET_FQDN>:8443/ | head -c 200
# Should return the orrery index.html shell — <!doctype html>...<title>Orrery</title>...

Browse: https://<PROD_TAILNET_FQDN>:8443/ — full app, with the live /data overlay if your cron has run pipeline-runner since deploy.

Rollback

What a deploy is made of decides how it rolls back. The workflow only runs on main (if: github.ref == 'refs/heads/main' on the deploy job), so dispatching it from an older tag, branch or commit does nothing: the job is skipped. Every deploy builds the static bundle and resets the VPS checkout from main's HEAD; only the three backend images can be pinned to another commit.

Part	Comes from	Roll back by
Static bundle (`/srv/orrery/build`)	built in the job from main's HEAD	a revert commit on main, then deploy
Server files (compose, Caddy drop-ins, `ops/`)	VPS `git reset --hard origin/main`	a revert commit on main, then deploy
`orrery-mcp`, `orrery-lab-api`, `orrery-pipeline-runner` images	GHCR `sha-<7>`, default main's HEAD	`override_image_sha=<earlier sha>`
Lab state (`/srv/orrery/lab-api-state`: tokens, allowlist, signing key)	the volume on the box	not touched by any deploy; nothing to roll back
Data overlay (`/srv/orrery/static/data`)	on-box refresh cron (RFC-035)	not touched by a deploy

Backend-only rollback (a bad mcp / lab-api image; the site is fine):

bash

# the last good commit's 7-char sha; its sha-<7> images must exist on GHCR
gh workflow run "Deploy to prod VPS (orrery)" --ref main --field override_image_sha=abc1234

The "Refuse a tag GHCR does not have" step fails the run before it touches the box if any of the three images is missing for that sha. The static bundle still comes from main's HEAD, so this only helps when the site itself is good.

Full rollback (the site or server files are bad):

bash

git revert <bad-commit>...        # on main; keeps history, no force-push
git push origin main              # CI + docker-e2e publish sha-<7> for the revert
# wait for CI + docker-e2e to go green on the revert commit, then:
gh workflow run "Deploy to prod VPS (orrery)" --ref main

Waiting for docker-e2e matters: it publishes the pipeline-runner sha-<7> image, and the deploy refuses a commit whose images are missing. If that wait is too long during an incident, deploy the revert with override_image_sha=<last good sha> so the images come from the last good commit while the bundle is built from the revert.

After any rollback:

The validate job in the same run executes npm run validate-prod against the public URL; it must be green.
Check the release on the box: ORRERY_RELEASE in /srv/orrery/.env is sha-<7> of the commit the workflow ran on (main's HEAD), not the override_image_sha.
Run the orrery-deploy-verify skill (.claude/skills/orrery-deploy-verify: logs, GlitchTip since the deploy, Umami events, source maps).

Staging (GitHub Pages, staging.yml) needs no rollback procedure of its own: it redeploys from main after every green docker-e2e, so the revert commit above fixes staging too. It can also be re-run by hand with gh workflow run "Deploy preview" --ref main.

Adding `push: branches: [main]` (graduation step)

After 3–5 successful manual deploys, add the auto-trigger by editing .github/workflows/deploy-prod.yml:

yaml

on:
  workflow_dispatch:
    inputs:
      override_image_sha:
        description: "..."
  workflow_run:
    workflows: [docker-e2e]    # gate: deploy only after docker-e2e green
    types: [completed]
    branches: [main]

Add a top-level job condition so the workflow_run leg only fires on success:

yaml

jobs:
  deploy:
    if: |
      github.event_name == 'workflow_dispatch'
      || github.event.workflow_run.conclusion == 'success'

This makes orrery's deploy chain: push to main → docker-e2e (mobile + desktop matrix + publish) → deploy-prod (auto).

Until then, every push needs a manual gh workflow run.

Troubleshooting

Workflow fails on "Missing prereqs"

The secrets/vars in §"GitHub Actions setup" aren't all set. Set them, re-run. (PROD_SSH_HOST_KEY is checked in the SSH step, not here: it fails that step instead of skipping the deploy.)

Re-pin the SSH host key

The SSH steps fail with Host key verification failed after the VPS gets new host keys, which every rebuild from Terraform/cloud-init does. Re-pin from two independent sources, and only if they agree:

On the box (prod session): ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub.
Over the tailnet (WireGuard-authenticated): ssh-keyscan -t ed25519 <PROD_TAILNET_FQDN> | ssh-keygen -lf -.
If the SHA256 fingerprints match, set the key (the part after the hostname): gh variable set PROD_SSH_HOST_KEY --env prod --body "ssh-ed25519 AAAA…".

Pinned 2026-10-17: SHA256:jcRHkLFcNnL122wb3nM7F5nNjEqCtyjfys6+sRJ1EA0 (the 2026-05-29 build).

Tailnet probe times out

Run on the VPS:

bash

sudo tailscale serve status
# Should list 8443 → http://127.0.0.1:8090

If :8443 isn't there, the systemd ExecStartPost didn't run. Inspect:

bash

sudo journalctl -u tailscaled.service --since "10 min ago" | grep -i orrery

If :8443 is registered but the probe still fails, the web container isn't responding on :8090. Inspect:

bash

cd /srv/orrery
docker compose -p orrery -f compose/docker-compose.prod.yml ps web
docker compose -p orrery -f compose/docker-compose.prod.yml logs --tail 30 web

Compose pull fails with `manifest unknown`

The pipeline-runner image hasn't been published yet. Verify:

bash

docker pull ghcr.io/chipi/orrery-pipeline-runner:main

If 404 — the docker-e2e publish job hasn't run successfully yet on main. Check Actions tab for a green run with both matrix legs (desktop + mobile) succeeded, then publish job ran.

Rsync 'permission denied' on /srv/orrery/build/

deploy@ user doesn't own the directory. On VPS:

bash

sudo chown -R deploy:deploy /srv/orrery/build /srv/orrery/static/data

What's NOT in phase 1

Public DNS / Let's Encrypt cert — phase 2 (#260 §"Phase 2").
Auto-deploy on push-to-main — defer until manual deploys have proven stable.
Grafana Cloud log shipping — retired (ADR-068 Superseded). Log shipping is active via shared node Alloy + ops/observability/orrery.alloy; no env vars needed.
Sentry — ADR-067 same story.
Backup of /srv/orrery/static/data — the data is derivable (every pipeline run rebuilds it). If derivability turns out to be slow, add a backup job pattern from podcast_scraper.

Briefs

Heroes

Observatories

Deploy orrery to prod VPS — operator runbook

Architecture

VPS prerequisites — owned by podcast_scraper cloud-init

The deploy flow

Caveat — tailscale-serve durability

GitHub Actions setup (one-time)

Required (deploy will skip with warning if missing)

Optional — observability (#263)

Triggering a deploy

Verifying

Rollback

Adding `push: branches: [main]` (graduation step)

Troubleshooting

Workflow fails on "Missing prereqs"

Re-pin the SSH host key

Tailnet probe times out

Compose pull fails with `manifest unknown`

Rsync 'permission denied' on /srv/orrery/build/

What's NOT in phase 1

Deploy orrery to prod VPS — operator runbook ​

Architecture ​

VPS prerequisites — owned by podcast_scraper cloud-init ​

The deploy flow ​

Caveat — tailscale-serve durability ​

GitHub Actions setup (one-time) ​

Required (deploy will skip with warning if missing) ​

Optional — observability (#263) ​

Triggering a deploy ​

Verifying ​

Rollback ​

Adding push: branches: [main] (graduation step) ​

Troubleshooting ​

Workflow fails on "Missing prereqs" ​

Re-pin the SSH host key ​

Tailnet probe times out ​

Compose pull fails with manifest unknown ​

Rsync 'permission denied' on /srv/orrery/build/ ​

What's NOT in phase 1 ​

Deploy orrery to prod VPS — operator runbook

Architecture

VPS prerequisites — owned by podcast_scraper cloud-init

The deploy flow

Caveat — tailscale-serve durability

GitHub Actions setup (one-time)

Required (deploy will skip with warning if missing)

Optional — observability (#263)

Triggering a deploy

Verifying

Rollback

Adding `push: branches: [main]` (graduation step)

Troubleshooting

Workflow fails on "Missing prereqs"

Re-pin the SSH host key

Tailnet probe times out

Compose pull fails with `manifest unknown`

Rsync 'permission denied' on /srv/orrery/build/

What's NOT in phase 1