ADR-065 — Live /data volume: pipelines write, web reads, no rebuild between refreshes

Status · Accepted Date · 2026-05-22 Closes into · RFC-024 §3, §5 Related ADRs · ADR-029 (PWA + service worker — NetworkFirst on /data/**/*.json), ADR-063 (web container), ADR-064 (pipeline runner), ADR-051 (CC-BY ship-gate + citation discipline) Closes follow-ups · "Pipelines write to a live /data volume the web container reads" — the data-flow decision from RFC-024's scoping pass

Context

The current production model is: pipelines run on GH Actions, commit refreshed JSON files to static/data/ on the main branch, and the standard CI deploy chain rebuilds + publishes to GH Pages. Every data refresh therefore ships as a git commit + a redeploy. This is fine for GH Pages (where the deploy is automatic and cheap) but doesn't generalise to "I want to refresh launches without rebuilding the entire site, and have the new data show up live."

RFC-024 introduces a docker-compose stack where the web container is always-on (no rebuild on data change) and pipelines are on-demand (run, write, exit). The seam between them is the data store. Three shapes were considered (RFC-024 scoping question): commit-and-redeploy (current behaviour, replicated in docker), live volume (no commits, web reads volume directly), hybrid (volume + commits for audit trail).

The user picked live volume. This ADR documents that choice with the constraints + contracts that make it work.

Decision

Use a single docker named volume orrery-data, mounted into the web container read-only at /usr/share/nginx/html/data and into the pipeline-runner container read-write at /repo/static/data. Pipelines write JSON files directly; nginx serves them on the next request. No git commits in the data-refresh path. The volume is treated as a regenerable cache, not a database.

Mount topology

• web: orrery-data:/usr/share/nginx/html/data:ro
• pipeline-runner: orrery-data:/repo/static/data (read-write)

The two mount paths resolve to the same volume content. Web's path is where adapter-static's /data/*.json URL contract lands inside the nginx image. Pipeline-runner's path is where scripts already write (static/data/launches.json, etc.) — the bind-mount inversion is invisible to the script because process.cwd() is /repo and the relative path static/data/launches.json resolves identically.

Initial seeding (first up on an empty volume)

On first mount of an empty named volume, docker copies the contents of the image's destination path into the volume. So:

1. docker compose build produces the web image with /usr/share/nginx/html/data/ populated by ADR-063's COPY --from=builder /repo/build /usr/share/nginx/html step — i.e. whatever static/data/ was committed in the repo at build time.
2. docker compose up -d web mounts the empty orrery-data volume at that path. Docker initialises the volume with the seed contents.
3. From that point on the seed is masked; pipeline writes update the volume in place; further image rebuilds don't overwrite the volume (this is docker named-volume semantics).

If an operator wants to reset the volume to the image's seed (e.g. discard a corrupted refresh), they run npm run docker:reset-data which docker volume rms both volumes; the next docker compose up re-seeds from the image.

Freshness contract

• Server side: nginx serves /data/*.json with Cache-Control: no-cache (ADR-063). The browser must revalidate; nginx serves whatever bytes are currently in the volume.
• Client side: the PWA service worker (ADR-029) uses NetworkFirst on /data/**/*.json with a 3-second timeout. Online users get the live volume contents; offline users get the cached copy.
• PWA install behaviour: the SW precaches the initial state of /data/*.json (because it's part of the asset manifest at build time). On subsequent visits, NetworkFirst strategy replaces the precached copy with the live fetch when online. New pipeline writes propagate within one fetch.

The combination of no-cache + NetworkFirst is what makes "pipeline writes → users see new data on next request" work without any explicit invalidation mechanism. The PWA SW already shipped this contract for the GH Pages model (where refreshes were just commits); the docker model uses the same plumbing.

What the volume DOES contain

Everything that today lives in static/data/ and is regenerable from a pipeline:

• launches.json + launches-historic/*.json (fetched by scripts/fetch-launches.ts)
• surface-hotspots.json + hotspot-metadata/* (fetched by scripts/fetch-hotspot-imagery.ts + curated)
• image-provenance.json (built by scripts/build-image-provenance.ts)
• link-provenance.json (built by scripts/build-link-provenance.ts)
• science-index.json (built by scripts/build-science-index.ts)
• All mission overlays, gallery manifests, porkchop precomputes, etc.

What the volume does NOT contain

• Source code. Lives in the image (web stage) or bind-mounted from the host (pipeline stage).
• User data. There is no user data — Orrery is a read-only static app.
• Generated bundle output. build/_app/immutable/ is in the image, not the volume.
• Schemas. static/data/schemas/*.json is in the image; schema bumps require an image rebuild, intentionally — they're a versioned contract.

Backup posture

The volume is not backed up. Every file in it is regenerable from a pipeline + source data (LL2, GCAT, HiRISE, NASA RSS, ESA RSS, etc.). If the volume is destroyed, npm run docker:reset-data && docker compose up -d web && npm run docker:fetch-launches && npm run docker:images:hotspots && … recovers state.

When the VPS RFC materialises, this posture may shift — production VPS likely wants nightly volume snapshots even though the data is regenerable, because pipeline re-runs against external sources take ~30 min total and a backup beats outage. Tracked under RFC-024 §10.

Schema versioning contract

If a pipeline's output schema changes (new required field, type rename, etc.), the image must be rebuilt alongside the pipeline change — otherwise an old web image reads new data shaped incompatibly with what the SvelteKit code expects. This is enforced by:

1. Validation: scripts/validate-data.ts runs inside the pipeline-runner before any write, against the schema in static/data/schemas/. If the schema in the bind-mounted source disagrees with the schema baked into the image, validation fails the same way it does today.
2. CI: docker-e2e.yml (ADR-066) rebuilds the image on every workflow run, so a schema change PR's CI run exercises the new image against the new pipeline output.

This is the same contract the GH Pages model has today (schema + pipeline + consumer all redeploy together); the volume model preserves it.

Rationale

A live volume is the only data-flow that meets all three of the user's stated constraints simultaneously: (a) pipelines runnable on demand, (b) web serves fresh data without rebuild, (c) no commit storm in git. Commit-and-rebuild violates (b) and (c); a database between pipeline + web violates the "regenerable from sources" simplicity that's load-bearing for this project.

Volume read-only on web side, read-write on pipeline side, is a defence-in-depth detail with zero behavioural impact (nginx never writes). It catches future regressions where someone accidentally adds a write-during-request path.

The PWA + NetworkFirst combo already does the right thing for freshness — we don't need a separate cache-bust mechanism, signed manifest, or "data version" header. The contract was set in ADR-029 (PWA) and ADR-045 (mission JSON refresh cycle) when the GH Pages model required the same property; we're inheriting a working pattern.

Treating the volume as a regenerable cache (not a database) is what lets us decline backup tooling at this stage. Every byte has a known upstream source; the cost of regeneration is bounded; the value of audit trail is zero (you can ask GCAT what the launch dates were on any past date). This is the property that justifies "no commit history for data refreshes."

Alternatives considered

• Commit-and-redeploy (current GH Pages model, replicated in docker). Mirrors today's behaviour exactly. But it requires the docker stack to have a git checkout + push capability, plus a way to trigger the web container to rebuild, plus a deploy pipeline. Three pieces of infrastructure to replicate something the live-volume model just doesn't need. Rejected as over-engineered for this scope.
• Hybrid: volume + git snapshot commits for audit. The audit-log property is plausibly useful (you could rewind to "what did launches look like on 2026-05-22?"). But it doubles the data-write code path (every script ends with volume write + git commit), introduces a permissions surface (push token in the pipeline container), and pulls back in the commit storm we're trying to avoid. The audit case is better served by archiving the manifest file periodically to S3-equivalent or by querying GCAT directly, both of which are outside this RFC. Rejected as over-engineered.
• Sqlite database file in the volume instead of JSON files. A real database would let us do partial updates without rewriting a whole launches.json. But the entire frontend reads JSON at HTTP, not SQL — switching the data plane would require server-side query handling at the web tier, which means abandoning nginx-only for adapter-node or a query API. Scope explosion. Rejected.
• Volume mounted into the host filesystem (bind mount) instead of named volume. Bind mount of ./static/data:/usr/share/nginx/html/data would let host git operations operate on pipeline output directly. Tempting for "I want to see what changed and commit it." But: (a) docker bind-mount permission semantics on macOS are notoriously inconsistent; (b) it couples pipeline behaviour to host filesystem layout; (c) it means the pipeline writes to the committed source, blurring the line between "regenerable cache" and "source of truth." Rejected; the operator who wants commit-style audit can docker compose cp web:/usr/share/nginx/html/data ./static/data on demand.
• Read-write volume on web side. Allowed in case nginx ever wants to write logs or generate sitemap.xml on the fly. Rejected: nginx doesn't need to, and read-only on the web side catches accidental regressions. Logs go to stdout (docker-managed); dynamic generation isn't in scope.

Consequences

Positive:

• Pipeline runs are fast and silent: no commit, no push, no rebuild. npm run docker:fetch-launches writes the new file; refresh the browser and you see it.
• The audit pipeline (scripts/audit-report-launches.ts) still works because it writes an HTML file into the same volume that the operator can view at http://localhost:8080/data/launches-audit.html.
• The "live data without rebuild" property is exactly what makes a future VPS deploy plausible — pipelines refresh data on a host crontab; the web container serves the new data on next request; no Docker-restart, no zero-downtime gymnastics.
• PWA + NetworkFirst already handles client-side freshness; we inherit a working pattern.

Negative:

• No audit trail for data changes. If a refresh produces a surprising number ("why does Atlas V 551 think it's first-flight now?"), the answer comes from re-running the pipeline + inspecting its logs, not from git log static/data/launches.json. The existing scripts/audit-report-launches.ts HTML output partially fills this gap.
• The volume is one of the things an operator has to know to reset (npm run docker:reset-data). Documented; not silent magic.
• Schema-change PRs have to remember to rebuild the image alongside the pipeline change. Enforced by docker-e2e.yml rebuilding on every workflow run; still a discipline point.
• The volume is host-local. A different host (or a teammate's laptop) has a different volume state. Acceptable for local dev (each contributor's volume is their own scratchpad); when the VPS comes online, the VPS's volume is the live data, which is the expected mental model.

Decision status

Accepted on 2026-05-22. Shipped in commit 4789855ee (Slice 1 of RFC-024) — with one deliberate divergence from the original Draft: the live data surface is a host bind-mount of ./static/data, NOT a docker named volume. Both the web container (./static/data:/usr/share/nginx/html/data:ro) and the pipeline-runner (./static/data:/repo/static/data rw) attach the same host path; nginx's /data/* URL contract resolves over the bind-mount, and pipeline writes propagate to the served bytes on the next HTTP request. The original "named volume with on-image seed" model was sound but redundant for local dev — the host repo IS the source of truth for static/data, and using a named volume would have introduced a copy-on-first-mount step that just duplicates what's already on disk. The named-volume model still applies for the future VPS RFC, where there's no host source tree; that RFC will introduce the seed-from-image bootstrap then.

Live-data property validated end-to-end on 2026-05-22: a host-side write to ./static/data/launches.json (canary key injection) showed up at http://localhost:8080/data/launches.json on the next curl, with the configured Cache-Control: no-cache header. Reverting the canary restored the original bytes immediately. The contract holds.

The orrery-cache named volume (.image-cache reuse for pipelines) IS still a named volume — pipelines write GB-scale binary blobs there and we don't want them in the host repo tree. That's a separate decision from the data-surface mount strategy and isn't affected by this divergence.