ADR-068 — Grafana Cloud Agent as docker-stack log shipper (profile-gated, env-var-gated, silent without credentials)

Status · Accepted Date · 2026-05-22 Closes into · RFC-025 §5, §8 Related ADRs · ADR-063 (web container), ADR-064 (pipeline runner), ADR-065 (live /data volume), ADR-066 (docker-e2e workflow) Reuses · podcast_scraper RFC-081 §Layer-2 Grafana Agent pattern (docker_sd_configs + remote_write to Grafana Cloud Loki, free tier, dashboards-as-code)

Context

RFC-024 landed a docker-compose stack — web (nginx serving the static bundle) + pipeline-runner (on-demand Node + tsx + system gdal). Today the stack runs only on contributors' laptops and inside the docker-e2e.yml CI runner. Tomorrow (per RFC-024 §10), the same stack runs on a VPS and the pipeline-runner is invoked on a host cron to refresh launches.json every 6 h. Once that VPS exists, three observability needs become real:

• nginx access logs. Who's hitting /data/*.json, how often, what's the status distribution? Useful for capacity planning + spotting CDN-cache-bust DDOS.
• Pipeline-runner invocation logs. Did the 6-hourly fetch-launches.ts cron succeed? How long did it take? Did it write new entries or were upstream sources unchanged?
• Process-level uptime. Is the web container restarting? Did docker reap a stuck pipeline-runner?

The VPS isn't here yet, but the integration shape — which Grafana product, how the agent ships, what dashboards we render, how credentials flow — is best designed when no real production traffic is at stake. Landing it now under env-var gates means the VPS RFC inherits a working integration instead of re-litigating it under deploy-stress.

The decision is the Grafana stack choice (cloud vs self-hosted), the log-shipping mechanism (Loki Promtail vs OTLP vs Docker driver), the credential flow (env vars vs compose secrets vs runtime injection), and the local-dev posture (silent default vs noisy default) — given Orrery's constraints:

• No committed secrets (per RFC-025 §3 + operator's explicit direction). Loki write URL, instance ID, API key all live in env vars.
• Default-off locally. Contributors running docker compose up shouldn't be shipping their laptop's logs anywhere. Has to be explicit opt-in.
• Reuse podcast_scraper's account. Operator has a Grafana Cloud free-tier stack already. Same stack hosts Orrery's data; no new billing, no new ops account.

Decision

Add a grafana-agent service to docker-compose.yml gated behind a new compose profile observability. Configuration lives in ops/observability/grafana-agent.yaml and tails the web + pipeline-runner containers' Docker logs via the docker_sd_configs discovery pattern. The Loki client block reads GRAFANA_CLOUD_LOKI_URL, GRAFANA_CLOUD_LOKI_USER, GRAFANA_CLOUD_API_KEY env vars; when any are unset the agent runs but ships nothing (logs tail to its own stdout only). Grafana dashboards live as JSON in ops/observability/dashboards/ and are committed; the operator imports them once into the Grafana Cloud org via a tiny import.sh wrapper around the Grafana HTTP API.

Service definition

# docker-compose.yml
grafana-agent:
  image: grafana/agent:v0.43.4
  profiles: ['observability']
  volumes:
    - ./ops/observability/grafana-agent.yaml:/etc/grafana-agent.yaml:ro
    - /var/run/docker.sock:/var/run/docker.sock:ro
  environment:
    - GRAFANA_CLOUD_LOKI_URL=${GRAFANA_CLOUD_LOKI_URL:-}
    - GRAFANA_CLOUD_LOKI_USER=${GRAFANA_CLOUD_LOKI_USER:-}
    - GRAFANA_CLOUD_API_KEY=${GRAFANA_CLOUD_API_KEY:-}
    - GRAFANA_AGENT_ENV=${GRAFANA_AGENT_ENV:-local-dev}
  command: ['-config.file=/etc/grafana-agent.yaml']
  restart: unless-stopped

profiles: ['observability'] is the linchpin — docker compose up -d web neither builds nor starts it. To run with observability locally:

docker compose --profile observability up -d

CI's docker-e2e.yml does NOT pass the profile flag, so the existing e2e gate (ADR-066) keeps observing only web.

Agent config — ops/observability/grafana-agent.yaml

server:
  log_level: warn

logs:
  configs:
    - name: orrery
      clients:
        - url: ${GRAFANA_CLOUD_LOKI_URL}
          basic_auth:
            username: ${GRAFANA_CLOUD_LOKI_USER}
            password: ${GRAFANA_CLOUD_API_KEY}
          external_labels:
            env: ${GRAFANA_AGENT_ENV}
            app: orrery
      positions:
        filename: /tmp/positions.yaml
      scrape_configs:
        - job_name: orrery-docker
          docker_sd_configs:
            - host: unix:///var/run/docker.sock
              refresh_interval: 30s
          relabel_configs:
            - source_labels: ['__meta_docker_container_name']
              regex: '/(orrery-web|orrery-pipeline-runner-.*)'
              action: keep
            - source_labels: ['__meta_docker_container_name']
              target_label: container
              regex: '/(.*)'
              replacement: '$1'
            - source_labels: ['__meta_docker_container_log_stream']
              target_label: stream

When GRAFANA_CLOUD_LOKI_URL is empty, the Agent's Loki client init treats the empty URL as "no remote target" and the log-shipping pipeline short-circuits to local-only stdout — no network calls. Verified by docker exec grafana-agent ss -tn showing zero established connections in the local-dev posture.

Dashboards as code — ops/observability/dashboards/

Two JSON files committed at Slice 2:

• orrery-web-access.json — nginx access volume over time (per status code), top requested paths, latency-percentile bucket (when nginx is configured with the right log format on the VPS — $request_time field).
• orrery-pipelines.json — pipeline-runner invocation count, error-line incidence (grep ^ERROR or panic: patterns), per-pipeline runtime distribution.

An import.sh script (also committed) wraps curl -X POST $GRAFANA_HTTP_URL/api/dashboards/import with the dashboards-as-code JSON. The operator runs it once per environment (local-dev, the future VPS); the dashboards live under the Grafana Cloud org's "Orrery" folder.

Dashboards are committed because they contain no secrets — just JSON describing panels, queries, and label selectors. They're operator playbook, not credentials.

Credential flow

Layer	Where the secret lives	Who sets it
Local dev	.env (gitignored)	Operator, once, optionally for testing
Future VPS production	Host env (systemd unit EnvironmentFile=, or compose --env-file)	Operator at VPS bring-up (future RFC)
CI (docker-e2e workflow)	NOT SET — workflow runs without the observability profile	Nobody — the gate doesn't ship logs

The .env.example is updated with GRAFANA_CLOUD_LOKI_URL, GRAFANA_CLOUD_LOKI_USER, GRAFANA_CLOUD_API_KEY, GRAFANA_AGENT_ENV rows — names + comments only, no values.

The scripts/check-no-secrets.ts scanner (introduced in ADR-067 / RFC-025 §3) covers Grafana Cloud API key patterns (glc_[A-Za-z0-9+/=]{40,}) — accidental commit blocked.

Rationale

Grafana Cloud free tier rather than self-hosted Loki/Prometheus: the operator already runs Grafana Cloud for podcast_scraper (per podcast_scraper RFC-081 §Layer-2). Same account, same Access Policies, same dashboards-as-code conventions. Self-hosted would mean adding Loki + Prometheus containers to the compose stack, running a separate Grafana UI container, and owning the storage rotation — operational burden for zero added capability vs the SaaS tier's 14-day / 50 GB / 10k-series limits, which Orrery's traffic will not approach.

Grafana Agent in Promtail-mode rather than the Docker loki log driver: the log driver is simpler but ties shipping to docker-engine config (host-level daemon.json), which doesn't compose well with the on-demand pipeline-runner pattern (the runner exits in seconds; a connection-pooled log driver doesn't help). Agent in Promtail-mode tails the Docker socket externally, sees container start + stop events, and ships logs at its own batching cadence.

docker_sd_configs over file-based tail of /var/lib/docker/containers/*/: the socket-discovery approach handles container churn cleanly (every new docker compose run --rm pipeline-runner … invocation gets discovered and shipped), and doesn't require us to mount the docker overlay storage. The trade-off — exposing the docker socket read-only to the agent — is the standard pattern for this kind of integration and the security model matches the operator-controlled host model (no third parties get inside the compose stack).

Profile-gated rather than always-on: contributors running docker compose up should not be shipping logs anywhere by default. The observability profile forces explicit opt-in. CI's docker-e2e.yml reaps the same property — it doesn't pass the profile, so the gate stays narrowly scoped to nginx-serving correctness.

Env-var-gated shipping rather than config-file-embedded credentials: same posture as ADR-067 and the operator's explicit direction. The compose service passes env vars through; the agent config substitutes ${VAR} references; empty values short-circuit the client init. The repo never contains a working credential.

Two starting dashboards rather than ten: scope discipline. The pipeline + nginx dashboards are the two views with concrete value once the VPS exists. Adding more dashboards before there's real data to populate them is shape-without-content.

Alternatives considered

• Self-hosted Loki + Grafana + Prometheus on the VPS. Full ownership, no SaaS dependency. Rejected — adds 3 containers, storage rotation, UI auth setup, and a parallel ops burden the operator already paid in podcast_scraper RFC-081 and explicitly chose not to repeat there.
• Docker loki log driver. Simpler. Rejected — host-config (/etc/docker/daemon.json) is global to the docker engine, doesn't isolate cleanly per-compose-stack, and the connection-pool model fits poorly with on-demand pipeline-runner containers that exit in seconds.
• OTLP exporter from a Node-side daemon. Modern, structured, vendor-neutral. Rejected for now — Orrery has no Node-side daemon to instrument (the pipeline-runner is one-shot, no persistent process). Revisit when the VPS RFC adds a longer-running scheduler.
• docker-compose.observability.yml override file. Alternative to compose profiles. Rejected — profiles are the docker-native way to express "service exists but isn't started by default," and the override-file pattern would require contributors to remember a second compose argument.
• Always-on agent with optional shipping. Agent runs by default but ships only when env vars set. Rejected — contributors don't expect a third service running locally if they didn't ask for observability. The profile-gate makes the intent visible: --profile observability means "yes, I'm doing observability work."
• Vector / Fluent Bit instead of Grafana Agent. Generic log shippers. Rejected — Grafana Agent is the official client for the Grafana Cloud Loki + Prometheus pair, with native config formats; using a third-party shipper adds translation friction without buying anything.
• No observability stack at all. Rely on docker compose logs web for triage. Rejected — that's fine for local dev (and remains the right tool there); on a VPS with 6-hourly cron'd pipelines and overnight access bursts, scrollback in a terminal isn't the right surface.
• Ship pipeline-runner stdout via a custom HTTP webhook (e.g. Discord, Slack). Operator alerting, not observability. Rejected because alerts ride on top of observability data; the prerequisite is having the data centralised somewhere queryable. Loki + dashboards is that somewhere; alerts on top are a future RFC.

Consequences

Positive:

• Observability shape is ready the moment the VPS RFC lands — operator populates env vars, runs docker compose --profile observability up -d, dashboards populate.
• Local dev contributors are unaffected unless they explicitly opt in.
• CI's docker-e2e gate is unaffected (no profile flag → no agent → no shipping).
• Reuses operator's existing Grafana Cloud account + free-tier quota.
• Two dashboards committed as code → infrastructure-as-code, importable + diffable + reviewable.

Negative:

• Adds ~50 LOC of compose + ~100 LOC of agent YAML + ~2 dashboard JSONs (~200 LOC each) for something that's silent today. Documented in this ADR + RFC-025 §5.4 as deliberate ("getting the shape right when nothing is live is the cheap moment").
• The grafana-agent image is ~250 MB once pulled. One-time cost per host; cached after first pull.
• Operator has to mount /var/run/docker.sock:ro into the agent. Read-only mount; standard pattern; security model is the same as Docker Desktop's own host control plane.
• A misconfigured grafana-agent.yaml (e.g. a scrape_configs regex that matches every container on the host, not just orrery-*) could ship unrelated logs to the operator's Grafana Cloud. Mitigated by the explicit keep action on __meta_docker_container_name matching /(orrery-web|orrery-pipeline-runner-.*). Verify on first VPS bring-up.

Decision status

Accepted on 2026-05-22. Shipped in commits da3d7423d (Slice 2 base) + 116378807 (entrypoint wrapper fix). The base contract held — Grafana Agent v0.43.4 in a profile-gated compose service, docker_sd_configs discovery filtered to orrery-* containers, env-var-gated Loki ship — but with one deliberate divergence from the original Draft documented in the Amendment — 2026-05-22 section below: instead of letting an empty GRAFANA_CLOUD_LOKI_URL short-circuit at the Loki-client init (which actually crashes the agent in a restart loop on "client needs target URL"), a small ops/observability/agent-entrypoint.sh shim picks between grafana-agent.yaml (shipping) and grafana-agent.silent.yaml (no-clients, no docker_sd) based on env-var presence.

Validated end-to-end on 2026-05-22:

• Empty creds (the local-dev default): docker logs orrery-grafana-agent emits one line Grafana Cloud creds NOT present → silent mode, container is stable (no restart loop), lsof -p $(docker inspect -f '{‌{.State.Pid}}' …) shows zero established outbound TCP connections.
• Populated creds (with a placeholder GRAFANA_CLOUD_API_KEY=glc_test_PLACEHOLDER_…): docker logs emits creds present → shipping config, agent enters Promtail/Loki client loop. (Auth fails against the fake API key as expected — the routing path is correct, the credentials aren't, which is the validation we wanted.)

Amendment — 2026-05-22 (during Slice 2 implementation): entrypoint shim picks silent vs shipping config

The original Draft said "agent stays running but the Loki client config short-circuits (no remote_write target) and the agent tails logs to its own stdout only — useful for 'is my logging output structured the way I expect?' but no external traffic." That description was aspirational, not measured. Reality: when GRAFANA_CLOUD_LOKI_URL expands to empty, the agent rejects the config at startup with client needs target URL and re-enters a restart loop every ~1-2 seconds, polluting docker logs and burning CPU.

The fix is structural: instead of one config that conditionally short-circuits at the client level, two configs and a tiny shim that picks based on env vars.

• ops/observability/grafana-agent.yaml — the shipping config (clients block populated by ${VAR} references, expanded at startup with -config.expand-env=true).
• ops/observability/grafana-agent.silent.yaml — a no-clients config (just server: { log_level: warn }, no logs subsystem). Loaded when creds are absent.
• ops/observability/agent-entrypoint.sh — sh-portable check: if all three creds set, exec the shipping path; else exec the silent path. Mounted into the container at /agent-entrypoint.sh and used as entrypoint:.

This preserves the contract this ADR documents (silent default → zero outbound) while making it operationally clean (no restart loop, one tidy log line on startup). The compose-side change is one block; the new shim is 17 lines of /bin/sh.

Also discovered during implementation: the agent binary inside grafana/agent:v0.43.4 is at /bin/grafana-agent, not /bin/agent as the v1 docs suggested. Entrypoint shim uses the correct path.

Amendment — 2026-05-29 (#263, grafana-agent lands in prod compose)

The original Slice 2 added grafana-agent only to the dev docker-compose.yml; prod compose/docker-compose.prod.yml explicitly omitted it per a phase-1 deferral note. #263 closes that gap.

Changes:

• grafana-agent service added to compose/docker-compose.prod.yml under profiles: ['observability']. Identical shape to the dev compose entry — same image, same bind-mounts for the three ops/observability configs + the docker socket, same env-var wiring.
• Default GRAFANA_AGENT_ENV=production-orrery (vs the dev default of local-dev) — distinguishes prod-orrery log streams from podcast_scraper's production-vps streams when both ship to the same Grafana Cloud Loki.
• Deploy workflow's docker compose up line passes --profile observability unconditionally. With creds empty, the silent-config shim takes over (per the 2026-05-22 amendment), agent runs idempotent without shipping. With creds present (staged into /srv/orrery/.env by the workflow's "Stage" step from GH prod env secrets), the shipping config picks up.
• GH prod env secrets used: PROD_GRAFANA_CLOUD_LOKI_URL, PROD_GRAFANA_CLOUD_LOKI_USER, PROD_GRAFANA_CLOUD_API_KEY. The workflow's Stage step strips the PROD_ prefix when writing to /srv/orrery/.env, so the agent reads them under their original names (matches ops/observability/grafana-agent.yaml's ${VAR} references).

Same Grafana Cloud stack as podcast_scraper is intentional — both apps' logs stream to one Loki, separated by the app label set in external_labels. Query orrery-only with {app="orrery"}; query podcast-scraper-only with {app="podcast-scraper"} (or whatever their agent labels with). One stack to monitor, one set of credentials to rotate.

The integration shape itself is stable. Once the VPS RFC ships, this ADR is revisited only if the dashboards or docker_sd_configs discovery filter need restructuring.