Skip to content

RFC-008 — Pipeline Parallelisation Strategy

Status · Decided · closed 2026-05-02 → ADR-023 TA anchor · pipeline/tasks · constraints Related · OC-04 six-pipeline-tasks · OC-02 creative-loop · RFC-009 (CLI) Closes into · ADR-023 Why this is an RFC · The daily pipeline and historical backfill both have parallelisable stages, but the right concurrency model — and its safe upper bound on this hardware — is not yet decided. Choosing the wrong model (e.g. threading for CPU-bound work, or uncapped Chromium instances) risks correctness failures or RAM exhaustion. The answer requires measuring actual render time and Chromium memory footprint before it can be locked.

The question

The current pipeline design is implicitly sequential: one source fetched, one source processed, one recipe rendered at a time. For the daily run with a small recipe count this is acceptable. For the historical backfill — up to 540 images at monthly cadence, or ~16,400 at daily cadence — sequential execution is the bottleneck that determines whether backfill is a one-hour job or a multi-day job.

The pipeline's six tasks split cleanly into two groups: tasks that are serial by dependency (Discover must precede Fetch; Index must follow all Renders), and tasks that are embarrassingly parallel within a stage (each source's Fetch is independent; each recipe's Render is independent). The question is how to exploit that independence safely and with what concurrency primitives, given this machine: an Apple MacBook Pro with many cores and 48 GB RAM.

Two sub-questions compound the main one. First, the right concurrency model differs by task type: I/O-bound work (Fetch) benefits from async or threading; CPU-bound work with subprocess isolation (Render via Puppeteer) benefits from process-level parallelism. Second, the safe upper bound on simultaneous Chromium instances is unknown until measured — too many and RAM pressure causes page-outs that make parallelism slower than sequential.

Use cases

Daily run. Multiple recipes accumulate over time. Rendering ten recipes sequentially at ~15s each = 2.5 minutes. Rendering them in parallel at a concurrency of 8 = ~20 seconds. The gap widens as recipe count grows.

Historical backfill. 540 monthly frames × ~12s = ~1.8 hours sequential. At concurrency 8 = ~14 minutes. This is the case that makes parallelism non-optional if the backfill is to be a practical one-session operation.

Recipe preview in editor. The live preview in the recipe editor is a single render triggered on demand. Parallelism does not apply here, but the render function's interface must be compatible with both single and batched invocation.

Goals

The parallelisation strategy must:

  • reduce backfill wall-clock time to under 30 minutes for the 540-image monthly dataset
  • keep peak RAM below 32 GB to leave 16 GB headroom for the OS and other processes
  • integrate with Prefect's existing @task / @flow model without replacing it
  • preserve Prefect's result caching — a date already rendered must never be re-rendered
  • leave the render(recipe, date) → png_path function signature pure and stateless, so it works identically in daily, backfill, and preview contexts

Constraints

From OC-04 constraints:

  • Orchestration is Prefect. The task runner is the correct insertion point for concurrency; the task functions themselves stay unchanged.
  • Deployment target is Docker Compose on this machine. No distributed workers, no Kubernetes.
  • Chromium instances are spawned by Puppeteer inside the pipeline container. Each instance carries an estimated 200–400 MB RSS. The actual figure must be measured in Slice 1.
  • The pipeline container runs Python + Node + Chromium. The concurrency limit must be a configurable constant, not hard-coded, so it can be tuned per host.

Proposed approach

Apply ConcurrentTaskRunner at the flow level and use Prefect's .submit() with a bounded semaphore for Task 05 (Render). Concurrency is controlled by a single constant RENDER_CONCURRENCY in pipeline config, defaulting to 6 pending measurement.

Chromium worker model

Each parallel render worker owns one persistent Chromium instance for the duration of its work. The instance is spawned once when the worker starts and kept alive across all renders assigned to that worker — load payload → render → screenshot → load next payload → repeat. The instance is closed only when the worker exits.

This eliminates per-frame Chromium startup cost (estimated 3–5s of a 12–15s render). The semaphore in the proposed approach caps workers, not individual render calls. RENDER_CONCURRENCY = 6 means six Chromium instances alive simultaneously, each processing its queue of assigned frames sequentially.

# Sketch — not implementation spec
RENDER_CONCURRENCY = 6  # caps workers (= Chromium instances), not frames

@flow(task_runner=ConcurrentTaskRunner())
def daily_pipeline():
    sources  = discover.map(ALL_SOURCES)
    fetched  = fetch.map(sources)
    processed = process.map(fetched)

    payloads = build_payload.map(ALL_RECIPES, wait_for=[processed])
    renders  = render.map(ALL_RECIPES, wait_for=[payloads])
    index.submit(wait_for=renders)

@flow(task_runner=ConcurrentTaskRunner())
def backfill_pipeline(recipe, start_date, end_date, cadence):
    dates   = generate_dates(start_date, end_date, cadence)
    renders = render.map([(recipe, d) for d in dates])
    index.submit(wait_for=renders)

Task-by-task breakdown

Task 02 — Fetch. Fan out across sources using .submit(). I/O-bound; threading is appropriate. Modest gain at three Phase 1 sources but costs nothing and scales as sources are added.

Task 03 — Process. Fan out across sources using .submit(). I/O-bound at the read step, briefly CPU-bound at the colormap step. Parallel across sources; per-date processing stays sequential within a source to keep file writes predictable.

Task 04 — Build Payload. Fan out across recipes using .submit(). Fast Python work; parallelism here eliminates the last sequential loop before render.

Task 05 — Render. Fan out across recipes (daily) or across dates (backfill) using .submit() with a semaphore capped at RENDER_CONCURRENCY. Each slot holds one persistent Chromium worker. Prefect result caching ensures already-rendered PNGs are skipped before the semaphore is acquired.

Task 06 — Index. Unchanged. Runs after all render futures resolve. Fast directory walk; no parallelism needed.

Alternatives considered

ProcessPoolExecutor outside Prefect. Would work for render parallelism but bypasses Prefect's result caching, retry logic, and observability. Every render would be opaque to the Prefect UI. Rejected: losing caching and retry is worse than any performance gain.

Async Puppeteer (Node worker threads). Running Puppeteer in async Node threads rather than separate Chromium processes would reduce overhead but introduces a Node concurrency model inside a Python task runner. Rejected at this scale; revisit if profiling shows Chromium inter-process overhead dominates.

Uncapped concurrency. Removing the semaphore entirely. With 540 backfill dates and fast .submit() calls this would spawn hundreds of Chromium instances simultaneously. At 300 MB/instance × 100 instances = 30 GB, saturating available memory. Rejected.

Docker child container workers. Spawning a worker container per render job, orchestrated from a parent container. Achieves the same parallelism as the internal model but adds Docker socket complexity, sibling container networking, and shared volume coordination. The internal ConcurrentTaskRunner + semaphore model achieves equivalent throughput on single-machine hardware without that surface area. Rejected.

Trade-offs

Parallel renders inside one container share the container's renders/ volume. File write contention is not expected — render output paths are non-overlapping per recipe per date — but must be confirmed during Slice 1 backfill testing.

ConcurrentTaskRunner uses Python threads, not processes. This is correct for I/O-bound tasks (Fetch) and for Render (where CPU work is inside Chromium subprocesses, not Python threads). If Task 03 (Process) proves slow under profiling, it should be moved to a ProcessPoolExecutor independently.

The persistent Chromium worker model means a crashed Chromium instance takes down its worker slot for the remainder of the run. Prefect's per-task timeout (timeout_seconds) mitigates runaway hangs; a crashed instance is logged and the slot is retired gracefully, with its unfinished frames failing and surfaced for re-run via oc render.

RENDER_CONCURRENCY must be measured, not guessed. The value of 6 is a starting point. Slice 1 empirical work sets the real ceiling.

Open questions

All design questions are resolved. Two items remain as Slice 1 empirical follow-ups, not blocking decisions:

  1. Chromium RSS in practice. Measure actual peak RSS per Chromium instance (field render vs particles render) during early Slice 1 using psutil or docker stats. Result sets the true safe ceiling for RENDER_CONCURRENCY. ADR-NNN (render concurrency limit) closes once this is measured.

  2. File write contention and semaphore placement. Confirm absence of filesystem contention under parallel writes during first backfill test. Confirm whether the semaphore lives more cleanly inside the task function (asyncio.Semaphore) or as a Prefect flow-level concurrency limit. Both are implementation details resolved by doing.

How this closes

ADR-NNN — Prefect task runner and concurrency model. Locks ConcurrentTaskRunner + .submit() and the persistent Chromium worker model. Closes early in Slice 1 once the implementation shape is confirmed.

ADR-NNN — Render concurrency limit. Locks RENDER_CONCURRENCY value once Chromium RSS is measured. Closes at end of Slice 1 after backfill smoke test.

  • OC-04 Technical Architecture — six-pipeline-tasks, constraints, tech-stack
  • OC-02 Project Concept — creative-loop (step 05 Accumulate)
  • RFC-009 — CLI Interface (backfill flow entry point: oc backfill)
  • Prefect docs — ConcurrentTaskRunner, .submit(), flow-level concurrency limits