ADR-064 — Pipeline runner: single tsx-entrypoint image, docker compose run --rm on-demand invocation, no long-running scheduler service

Status · Accepted Date · 2026-05-22 Closes into · RFC-024 §4, §5, §6 Related ADRs · ADR-014 (GH Pages deploy), ADR-063 (web container) Closes follow-ups · The "all pipelines I want to be able to run as docker process on demand" requirement from the RFC-024 scoping pass

Context

Orrery has ~10 data pipelines exposed as npm run … scripts: fetch-launches, images:hotspots, fetch-assets, build-image-provenance, build-link-provenance, validate-data, build-science-index, build-tech-bom, precompute-porkchops, check-learn-links. Today each runs either on a contributor's host (with whatever system gdal/sharp version that machine has) or on a GH Actions runner (refresh-launches.yml, ci.yml, release.yml).

RFC-024 wants every one of these runnable as a docker process so (a) contributors don't need a working gdal-async build chain on their laptop; (b) the eventual VPS can host pipeline runs that don't fit GH Actions cron windows; (c) the local + CI artefacts that pipelines produce are bit-identical to what production will produce.

The decision is the invocation pattern: how does the operator (or CI step) reach a script inside a container, what's the entry contract, and is there a long-running service in the mix or only on-demand containers? Three shapes were on the table — long-running pipeline daemon, on-demand docker compose run, or sidecar-with-internal-cron. RFC-024 §6 settled on on-demand.

Decision

One image — the pipeline stage of the multi-stage Dockerfile in ADR-063 — carrying Node 20 + tsx + system gdal + the locked node_modules. ENTRYPOINT is ["npx","tsx"]. The compose service pipeline-runner is declared with profiles: ['manual'] so docker compose up does NOT start it. Operators invoke pipelines via docker compose run --rm pipeline-runner <script.ts> [args], which spawns a one-shot container that exits when the script does. package.json provides short docker:<pipeline-name> wrappers so the CLI surface matches the existing npm run … form.

Image content

FROM node:20-bookworm-slim AS pipeline
RUN apt-get update && apt-get install -y --no-install-recommends \
      libgdal-dev gdal-bin \
      ca-certificates curl \
    && rm -rf /var/lib/apt/lists/*
WORKDIR /repo
COPY package.json package-lock.json ./
RUN npm ci          # installs gdal-async against system gdal
ENTRYPOINT ["npx", "tsx"]

Image carries:

• Node 20 + npm (matches .github/workflows/ci.yml).
• System gdal (libgdal-dev + gdal-bin) so gdal-async resolves at runtime.
• The full node_modules from package-lock.json. No source code.

Image does not carry repo source — that's bind-mounted from the host at runtime (volumes: [.:/repo:ro]). This is the deliberate trade-off for local dev: a script edit doesn't require a container rebuild.

Compose entry

pipeline-runner:
  build:
    context: .
    target: pipeline
  profiles: ['manual']
  volumes:
    - orrery-data:/repo/static/data        # rw — pipelines write here
    - orrery-cache:/repo/.image-cache      # rw — caches for HiRISE JP2s, GCAT TSV, LL2 responses
    - .:/repo:ro                           # ro — source code for tsx
  environment:
    - LL2_PATREON_KEY=${LL2_PATREON_KEY:-}
    - NODE_OPTIONS=--max-old-space-size=4096

profiles: ['manual'] is the linchpin. With it, docker compose up only brings up services without a profile (i.e. web). The pipeline-runner only exists when the operator types docker compose run --rm pipeline-runner ….

--rm is the second linchpin. It deletes the one-shot container on exit. Without it, docker ps -a accumulates stopped containers from every pipeline invocation.

Invocation surface

# Long form — what the wrapper scripts shell out to
docker compose run --rm pipeline-runner scripts/fetch-launches.ts

# Short form — what humans use
npm run docker:fetch-launches

The npm wrapper layer is in package.json. Every pipeline gets a docker:<name> script that calls the long form. The wrapper layer is the API; the long form is the mechanism.

Arguments and env

Pipeline args pass through transparently: docker compose run --rm pipeline-runner scripts/fetch-launches.ts --since 2026-05-01 works because the tsx entrypoint takes the script path + args verbatim.

Pipelines that need secrets read them from a repo-root .env (gitignored). Compose loads this automatically into pipeline-runner's environment. CI sets them directly via ${‌{ secrets.* }} (GH Actions secret-context syntax). Same shape as the existing refresh-launches.yml.

What we are NOT doing

• No long-running pipeline service. Some Compose-based projects run a "worker" daemon that loops, polls a queue, and processes jobs. Out of scope. Pipelines here are batch — they ingest, transform, write, exit. A daemon adds restart-policy reasoning, healthcheck reasoning, and a lifecycle we don't need.
• No internal cron container. Could schedule refreshes inside the stack via an alpine + crond sidecar. Out of scope for this RFC. The host crontab (or systemd.timer once we get to a VPS) is the obvious place; refresh-launches.yml is the obvious place today. Adding a third scheduler would compete with both.
• No tsx watch or hot-reload entrypoint. Pipelines run once, write the artefact, exit. Re-invoking is cheap (~2-5s overhead).
• No exposed network port on pipeline-runner. Nothing connects to it. It's not a service.
• No HTTP/RPC pipeline trigger API. The "trigger" is docker compose run. If we want webhooks (e.g. "GitHub push → run images:hotspots refresh") later, that's a separate component, not a property of the pipeline image.

Rationale

On-demand docker compose run --rm is the simplest invocation shape that:

• Maps 1:1 with the existing npm run <script> surface (operators don't have to relearn).
• Avoids a lifecycle for the pipeline tier (no restart policies, no healthchecks, no "pipeline service crashed at 3am" pages).
• Composes cleanly with CI (the same one-liner works in a workflow step).
• Naturally serialises — two docker compose run --rm pipeline-runner … invocations are independent processes; they share the data volume but don't deadlock or stomp on each other unless the scripts themselves race (which is an existing property of the scripts, not introduced by docker).

profiles: ['manual'] is the docker-native way to declare "this service exists but isn't started by up." Without it, docker compose up would start the pipeline-runner with no entrypoint args (or with ["npx","tsx"] and nothing to run), and it'd exit immediately with code 0, generating noise. With profiles, the service is latent until explicitly invoked.

Source bind-mount (.:/repo:ro) is the right trade-off for local + CI:

• A script edit doesn't require docker compose build. The next npm run docker:fetch-launches picks up the change instantly.
• CI checkout populates the bind-mount with the current ref's source, so the same image runs main-ref scripts on main PRs and feature-branch scripts on feature PRs without rebuilds.
• The trade-off: in the future VPS deploy the bind-mount goes away (no host repo on a VPS), and the pipeline image bakes source in. That's a known follow-up in RFC-024 §10; doesn't change the contract here.

node:20-bookworm-slim matches ADR-063's builder stage (same Node version → same node_modules resolution → no Node-version-skew bugs). System gdal install is a one-time apt-get; subsequent rebuilds hit cache as long as package-lock.json doesn't move.

ENTRYPOINT = ["npx","tsx"] rather than ["bash"] or ["node"] because:

• tsx is what every pipeline script uses already (tsx scripts/fetch-launches.ts).
• Operators don't have to remember npx tsx scripts/… — they say docker compose run --rm pipeline-runner scripts/… and the entrypoint adds the rest.
• Forbids accidental misuse (shell-escapes inside the container).

Alternatives considered

• Long-running pipeline daemon with a queue. Adds Redis (or sqlite, or filesystem queue). Lifecycle, restart policy, healthcheck. Solves a problem we don't have. Rejected.
• Cron container inside the stack (alpine + crond + script invocations). Compete with refresh-launches.yml. Adds a piece of infrastructure whose only job is invoking the pipeline image — and docker compose run from a host cron is the same thing minus a container. Rejected for the local-dev RFC; revisit in the VPS RFC.
• Separate images for separate pipelines (one for launches, one for hotspots, one for vision). Each could be smaller (no gdal in the launches image, etc.). Adds 5-10 Dockerfiles and a build-matrix. The shared node_modules is the bulk of every image; specialising base layers wouldn't recover the size cost. Rejected as premature splitting.
• Run pipelines on the host (no container) but use the data volume on disk. Defeats the point — contributors still need a working gdal/sharp build chain. Reproducibility argument disappears.
• ENTRYPOINT = ["bash"] with operators typing docker compose run --rm pipeline-runner bash -lc 'npx tsx scripts/…'. Longer, easier to typo, no real benefit. Rejected.
• GH Actions remote-trigger / gh workflow run as the "on demand" mechanism instead of docker. Couples local dev to GH Actions runner availability + minutes. Doesn't solve the contributor-without-gdal problem. Rejected.

Consequences

Positive:

• Any contributor can run any pipeline against the live data volume with one npm run docker:… command. No host gdal install. No "works on my mac but not your mac."
• CI workflows can re-use the same invocation pattern verbatim (docker compose run --rm pipeline-runner scripts/…), so a workflow's pipeline step matches what an operator runs locally.
• One-shot containers + --rm means there's no accumulating cruft in docker ps -a and no orphaned named containers blocking re-runs.
• The path to VPS is "swap the bind-mount for a baked-in source COPY" — a one-line Dockerfile change in the future compose.prod.yml overlay, no architectural reshape.

Negative:

• Per-pipeline cold-start cost: spinning up a container adds ~1-2s on top of the script. Acceptable for batch jobs that run in seconds-to-minutes; would be annoying if these were < 1s pings, which they aren't.
• The single image carries gdal + sharp + all node_modules even for pipelines that don't need them (e.g. build-science-index.ts is pure Markdown wrangling). Image is ~1 GB. Acceptable trade-off vs the alternative of multi-image complexity; the host has the bytes.
• Bind-mounting host source means pipeline determinism depends on the host's working tree. CI is fine (actions/checkout); local invocations against a dirty tree run dirty-tree scripts. This is correct behaviour for dev, not a defect.
• Operators who type docker compose up will see only web start — they have to know pipeline-runner is profiles: ['manual'] and uses run --rm. Documented in docs/guides/docker-stack.md and in --help output of the npm run docker:* wrappers.

Decision status

Accepted on 2026-05-22. Shipped in commit 4789855ee (Slice 1 of RFC-024). The Dockerfile is now repo-root single-stage (no longer a multi-stage with web as the secondary target — see ADR-063's Amendment for why) and contains only the pipeline-runner image. Verified: docker compose config pipeline-runner parses cleanly, docker build --check returns no warnings, and the profiles: ['manual'] gate means docker compose up -d web neither builds nor starts it. The image only builds on first docker compose run --rm pipeline-runner … — operators don't pay the ~5-10 min one-time gdal/sharp/canvas compile cost just to serve the static bundle locally.