RFC-009 — Pipeline CLI Interface

Status · Decided · closed 2026-05-02 → ADR-022 TA anchor · components/pipeline · constraints Related · OC-04 six-pipeline-tasks · OC-04 deployment · RFC-008 (parallelisation) Closes into · ADR-022 Why this is an RFC · The pipeline currently has no defined entry point for ad-hoc invocation. The Prefect server handles the scheduled daily run, but there is no sanctioned way to trigger a backfill, re-render a single date, or inspect pipeline state without going through Docker internals or the Prefect UI directly. Adding a CLI requires deciding how it relates to the Prefect server — whether it calls flows directly, dispatches through the server, or both — and that relationship determines how observability is split across invocation modes. This is a genuine design question with real trade-offs between simplicity and visibility.

The question

The pipeline runs on a daily schedule via Prefect Server inside Docker Compose. That covers the production cadence, but it leaves a gap: there is no clean way to invoke the pipeline outside that schedule. Triggering a backfill, re-rendering a failed date, validating a new recipe, or inspecting what has and has not been rendered all require either navigating the Prefect UI, running docker compose exec commands, or writing ad-hoc scripts.

A CLI closes that gap. The design question is how the CLI relates to the Prefect Server. Prefect flows can be invoked in two modes: directly as Python functions (the @flow decorator works without a running server — caching and retries still operate via the filesystem), or dispatched to the server (the run appears in the UI, is logged to Prefect's database, and is observable in the dashboard at localhost:4200).

These two modes have different characteristics. Direct invocation is simpler, always available, and gives immediate terminal feedback. Server dispatch provides structured run history, task-level timing, and error traces in the Prefect UI — the observability layer that makes the daily production run legible over time. The RFC proposes a design that uses both, with the invocation mode as an explicit flag rather than a hidden implementation detail.

Use cases

Backfill. Running the 45-year historical dataset (RFC-008) is the primary motivation. The user needs to invoke backfill with a recipe name, date range, and cadence, then watch progress in the terminal. This is a long-running job — up to 30 minutes for 540 frames — that does not need the Prefect UI but does need structured terminal output so the user can see which frames have completed and which have failed.

Single render. After editing a recipe YAML and wanting to see the result immediately, the user runs a single render for today's date without triggering the full pipeline (no fetch, no process — just payload build and render). This is the tightest feedback loop in the creative workflow.

Recipe validation. Before committing a new recipe, the user validates it: schema check, region sanity check, source availability check. Currently no tooling exists for this.

Status inspection. The user wants to know: which recipes have been rendered, what date range each covers, what is missing, and when the last successful run was. This is a read-only operation that should work without the Prefect server running.

Production run management. The scheduled daily run goes through the Prefect server and is visible in the UI. The user may want to trigger an out-of-schedule run through the server (e.g. after a source was unavailable at 06:00 UTC and has since recovered) and have it appear in the run history alongside the scheduled runs.

Goals

The CLI must:

provide a clean oc entry point usable from any terminal session, inside or outside Docker
cover the full operational surface: run, backfill, render, status, recipes, index
distinguish between direct invocation (always available, terminal output) and server dispatch (requires running stack, UI visibility) via an explicit --via-server flag
stream structured progress to the terminal for long-running operations (backfill)
work without the Prefect server running for all read operations and direct-invocation commands
integrate cleanly with RFC-008's backfill flow — oc backfill is the primary entry point for historical renders
be declared in pyproject.toml as a console script so oc is available as a command after pip install -e .

Constraints

From OC-04 constraints and deployment:

The pipeline container runs Python + Node + Chromium inside Docker Compose. The CLI runs inside this container; docker compose exec pipeline oc <command> is the invocation pattern from the host.
Alternatively, if the CLI is installed in the host Python environment (outside Docker), it can connect to the pipeline's file system via mounted volumes. This is a secondary invocation path, not the primary.
Prefect Server runs as a separate container in Docker Compose at localhost:4200. It is always running when the stack is up but must not be a hard dependency for direct-invocation commands.
CLAUDE.md must document which entry point is canonical for each use case (scheduled run vs ad-hoc run vs backfill) to prevent two invocation paths producing divergent pipeline state.

Proposed approach

A typer-based CLI installed as the oc console script. Two invocation modes unified under one interface, with --via-server as the explicit switch for server dispatch. The scheduled daily run continues to go through the Prefect server as before — the CLI does not replace or interfere with it.

Command surface

oc run [--via-server] [--dry-run]
    Run the full daily pipeline for today's date.
    Default: direct invocation, output to terminal.
    --via-server: dispatch to Prefect Server, visible in UI at localhost:4200.
    --dry-run: resolve what would run without executing.

oc backfill --recipe NAME --from DATE [--to DATE] [--cadence monthly|daily]
             [--concurrency N] [--via-server]
    Run the backfill flow for a recipe over a date range.
    DATE format: YYYY-MM or YYYY-MM-DD depending on cadence.
    --concurrency: override RENDER_CONCURRENCY for this run (RFC-008).
    Streams per-frame progress to terminal. Skips already-rendered dates
    (Prefect cache). Exits with non-zero code if any frame fails.

oc render --recipe NAME --date DATE [--force]
    Render a single frame. Skips fetch and process — reads from
    data/processed/ directly. Fails fast if processed data is absent.
    --force: re-render even if renders/{recipe}/{date}.png exists.
    Primary use: recipe iteration during the creative loop.

oc status [--recipe NAME]
    Print pipeline state to terminal. No server required.
    Without --recipe: summary table of all recipes — frame count,
    date range, last render timestamp, missing-date gaps.
    With --recipe: detailed view for one recipe including gap analysis.

oc recipes list
    List all recipes in recipes/ with name, render type, source, region.

oc recipes validate NAME
    Validate a recipe YAML: schema check, lat/lon sanity, source
    availability check against data/processed/. Prints pass/fail per check.

oc index --rebuild
    Manually rebuild manifest.json from renders/. Idempotent.
    Use after manual file operations or a partial pipeline failure.

Invocation modes in detail

Direct invocation (default)

oc run calls daily_pipeline() as a plain Python function. Prefect's @task decorator still applies — result caching operates on the filesystem (data/processed/ cache hits are respected), retries fire on network failure. The run does not appear in the Prefect UI. Output is streamed to the terminal with a simple structured format:

[06:14:22] discover     ✓  3 sources · latest dates resolved
[06:14:23] fetch        ✓  oisst 2026-04-30 · 2.1 MB
[06:14:31] process      ✓  oisst · 720×1440 grid · 0.3% NaN
[06:14:31] build        ✓  north_atlantic_sst · gulf_stream_thermal
[06:14:33] render       ✓  north_atlantic_sst · 11.2s
[06:14:46] render       ✓  gulf_stream_thermal · 13.0s
[06:14:46] index        ✓  manifest.json · 2 recipes · 127 frames total

Server dispatch (--via-server)

oc run --via-server submits a flow run to the Prefect Server via the Prefect client API. The run appears in the UI at localhost:4200, is logged to Prefect's database, and contributes to run history alongside the scheduled daily runs. The terminal shows the flow run ID and a URL to the run detail page:

Flow run submitted → http://localhost:4200/flow-runs/abc123
Watching run... (Ctrl+C to detach, run continues in server)
[06:14:22] discover  ✓
[06:14:31] fetch     ✓
...

Scheduled daily run (unchanged)

The 06:00 UTC daily run is declared as a Prefect deployment in Docker Compose. It goes through the server, appears in run history, and is unaffected by the CLI. This is the production path. The CLI's oc run --via-server produces an identical run via the same deployment, just triggered on demand rather than on schedule.

Terminal output for backfill

Backfill is the only long-running command. Progress is streamed as a compact per-frame line, with a live summary bar:

oc backfill --recipe gulf_stream_thermal --from 1981-01 --cadence monthly

Backfill · gulf_stream_thermal · 1981-01 → 2026-04 · 544 frames · concurrency 6

[████████████░░░░░░░░░░░░░░░░░░] 203/544  37%  ~18m remaining

✓ 1981-01   8.9s      ✓ 1981-02   9.1s      ✓ 1981-03  11.2s
✓ 1981-04   9.8s      ✓ 1981-05   8.7s      ✗ 1981-06  FAILED
✓ 1981-07  10.1s      ...

Completed: 202   Failed: 1   Skipped (cached): 0
Failed dates: 1981-06  →  re-run with: oc render --recipe gulf_stream_thermal --date 1981-06-01

Failed frames are surfaced immediately with the exact re-run command, so fixing a single failed render requires one follow-up invocation rather than re-running the entire backfill.

Library and installation

typer with rich output (rich library for progress bars and coloured terminal output). Declared in pyproject.toml:

[project.scripts]
oc = "oceancanvas.cli:app"

[project.optional-dependencies]
cli = ["typer[all]>=0.12", "rich>=13"]

Installed inside the pipeline container at build time. Available on the host if the project is installed with pip install -e ".[cli]".

Alternatives considered

Makefile targets instead of a CLI. make run, make backfill DATE=1981-01 etc. Simple, zero dependencies. Rejected: Makefiles do not compose well (no --help, no argument validation, no progress streaming), are awkward to call from inside Docker, and produce no structured output. A CLI is the right abstraction at this project's complexity level.

Prefect's own CLI (prefect run flow --name ...). Prefect ships a CLI that can trigger flow runs against the server. Rejected as the primary interface: it requires the server to be running for all invocations, has no knowledge of OceanCanvas-specific concepts (recipes, backfill cadence, gap analysis), and cannot do direct invocation. It remains available as a power-user escape hatch.

Click instead of typer. Click is the underlying library typer wraps. Rejected in favour of typer: typer's type-annotation-driven argument parsing eliminates boilerplate, produces better --help output automatically, and handles the --via-server flag pattern cleanly. The only cost is a thin wrapper dependency; acceptable.

Always dispatch through the server (no direct invocation). Simpler mental model — one invocation path, everything visible in the UI. Rejected: this makes the backfill and single-render commands dependent on the Prefect server being healthy, adds latency to every invocation, and means oc render --recipe X --date Y cannot be used when iterating quickly on a recipe with the full Docker stack down. Direct invocation must be the default.

Trade-offs

Direct invocation bypasses Prefect Server's run database. Ad-hoc oc run and oc backfill executions are not recorded in the UI's run history. This is acceptable because the primary historical record — the daily production run — goes through the server. Ad-hoc runs are operational actions, not production events.

Two invocation paths (direct and server dispatch) must produce identical output — the same PNGs in the same locations, the same manifest.json. The risk is drift: a bug in the server-dispatch path that does not appear in direct invocation, or vice versa. Mitigation: both paths call the same Prefect flow functions. The only difference is the Python call site. The e2e CI test (GitHub Actions) runs in direct invocation mode; a separate smoke test should confirm server dispatch reaches the same result.

The --via-server flag name must be documented clearly in CLAUDE.md to prevent a future developer from treating direct invocation as the "real" pipeline and server dispatch as an optional extra. The production daily run goes through the server; the flag exists to bring ad-hoc runs into that same observable record when needed.

Open questions

Host vs container installation. Should oc be installable on the host (outside Docker) and connect to the pipeline's volumes via mounts? This would allow oc status without docker compose exec. The volume mount paths would need to be configurable. Defer to implementation — start with container-only, add host support if it proves useful.
oc render and missing processed data. If oc render --recipe X --date 1985-03 is called but data/processed/oisst/1985-03-01.json does not exist, the command must fail clearly. Should it offer to fetch-and-process first, or fail-fast and tell the user to run oc run --date 1985-03 first? Lean toward fail-fast with a clear message — auto-fetching in a render command is surprising behaviour.
oc status gap analysis. Detecting missing dates requires knowing the expected date range per recipe. The recipe YAML does not currently declare a start_date. Should it? Without it, gap analysis can only infer the expected range from the first and last rendered frames, which is approximate. This may be a recipe schema question (RFC-001).
Colour and rich output in Docker logs. rich output with ANSI colour codes looks correct in an interactive terminal but produces noise in Docker log output. The CLI should detect TTY attachment and fall back to plain text in non-interactive contexts (e.g. when oc run is called from a cron job or CI).

How this closes

ADR-NNN — CLI entry point and invocation model. Locks typer as the CLI library, oc as the console script name, and the direct/server-dispatch dual-mode design. Closes once the command surface is agreed and the pyproject.toml entry is committed.

ADR-NNN — Prefect server role clarification. Locks the definition: Prefect Server handles the scheduled daily run and server-dispatch invocations; direct invocation is the default for all ad-hoc CLI commands. Updates CLAUDE.md with the canonical entry point per use case. Closes alongside the ADR above.

Links

OC-04 Technical Architecture — six-pipeline-tasks, deployment, tech-stack
RFC-008 — Pipeline Parallelisation Strategy (backfill flow, RENDER_CONCURRENCY)
RFC-001 — Recipe YAML schema (open question 3: start_date field)
Prefect docs — flow invocation without server, client API for server dispatch
typer docs — console scripts, argument validation, rich integration