Prod observability control plane (`podcast_obs`, #803)¶

A small, standalone control plane that answers "what is a deploy doing right now?" — health, version, recent pipeline runs and deploys, today's LLM cost, recent error logs (Loki) and Sentry issues, and current Grafana alerts — for any deploy (your local stack, prod over Tailscale, a drill). It's target-agnostic and degrades gracefully: a source that isn't wired for a target just reports configured=false.

It lives in its own light package (podcast_obs) with zero coupling to the heavy podcast_scraper pipeline, so it runs cheaply as a plain process or a small container anywhere on the tailnet.

Layers¶

Layer	What	Entry point
Core	probe/aggregate functions, one per backing system	`podcast_obs.sources.*`
CLI ("the basics")	probe any deploy directly, scriptable, no MCP	`python -m podcast_obs <cmd>`
MCP server	the same probes as agent tools	`python -m podcast_obs serve`
Docker	the standalone container control plane	`docker/observability/`

Install¶

httpx + PyYAML are base deps; the agent-facing layer adds mcp:

pip install -e '.[observability]'   # in this repo

For a slim standalone deploy, the container installs only httpx/PyYAML/mcp (see Docker below).

Configure¶

Target-agnostic. Either a single target from env vars, or multiple targets from YAML.

Env (single target): prefix PODCAST_OBS_.

Var	Used by	Notes
`PODCAST_OBS_API_BASE`	health/version/runs	e.g. `http://localhost:8080` or `https://prod-podcast.<tailnet>`
`PODCAST_OBS_GITHUB_REPO` / `_GITHUB_TOKEN`	deploys	default repo `chipi/podcast_scraper`
`PODCAST_OBS_SENTRY_ORG` / `_SENTRY_PROJECTS` / `_SENTRY_TOKEN` / `_SENTRY_ENV`	errors	projects CSV; env default `prod`
`PODCAST_OBS_GRAFANA_URL` / `_GRAFANA_TOKEN`	alerts (+ Loki token)	Grafana stack URL
`PODCAST_OBS_LOKI_URL` / `_LOKI_USER`	cost/logs	Loki push or query URL (suffix is normalised)
`PODCAST_OBS_ENV_LABEL`	cost/logs	the deploy's Loki `env` label (default `prod`)
`PODCAST_OBS_TIMEOUT`	all	per-request seconds (default 10)

YAML (multi-target): point PODCAST_OBS_CONFIG at a file. Secrets stay out of the file via <field>_env: indirection (the value is read from the named env var). See config/observability.example.yaml:

default_target: local
targets:
  local:
    api_base: http://localhost:8080
  prod:
    api_base: https://prod-podcast.tail-xxxxx.ts.net
    env_label: prod
    github:
      repo: chipi/podcast_scraper
      token_env: PODCAST_OBS_GH_TOKEN
    sentry:
      org: your-org
      projects: [api, pipeline, viewer]   # real slugs (Settings → Projects), not DSN names
      token_env: PODCAST_OBS_SENTRY_TOKEN
    grafana:
      url: http://homelab:3000
      token_env: PODCAST_OBS_GRAFANA_TOKEN      # service-account token (glsa_) for alerting
      loki_url: http://homelab:9428
      loki_token_env: PODCAST_OBS_LOKI_TOKEN    # VictoriaLogs read token (if auth enabled)

Tokens and scopes (read-only by default; the two write tools are gated, see MCP section)¶

Source	Credential	Scope / gotcha
`prod_api`	none	Reachability only (tailnet-gated in prod).
`github`	fine-grained PAT, or the `gh` CLI	Actions: read.
`sentry`	a Sentry auth token	Issue & Event: Read (`event:read`) — `project:read` alone is not enough. NOT the DSN (the staged `PROD_SENTRY_DSN_` can't query the API). `prod_recent_errors`/D2 also want Release: Admin. Note: project slugs* ≠ DSN names (check Settings → Projects; e.g. `podcast-scraper-api`).
`grafana` (alerts)	a Grafana service-account token (`glsa_`)	alerting read. Grafana-API only (homelab:3000).
`loki` (cost/logs)	VictoriaLogs read token (if auth configured)	`logs:read` against homelab:9428. If VictoriaLogs runs without auth (default self-hosted), this can be omitted — the grafana token suffices for the Grafana API surface. Historical note: Grafana Cloud split the data plane (Loki `glc_`) from the Grafana API (`glsa_`) — self-hosted does not require this split.
`langfuse` (traces)	a Langfuse public + secret key pair (Basic auth)	Read-only public API. SDK-native bare env names `LANGFUSE_PUBLIC_KEY` / `LANGFUSE_SECRET_KEY` (not `PODCAST_OBS_`) — the same* pair the pipeline traces with, so one set drives both emit + probe. Always set `LANGFUSE_BASE_URL=http://homelab:4000` (self-hosted); if unset the SDK silently defaults to Langfuse Cloud (billed per event, off-tailnet) — this estate is self-hosted (ADR-0005).

CLI (the basics)¶

python -m podcast_obs summary    --target prod      # control-plane glance, all sources
python -m podcast_obs health     --target local
python -m podcast_obs version    --target prod
python -m podcast_obs runs        --limit 5         # recent pipeline runs (/api/jobs)
python -m podcast_obs deploys     --limit 5         # deploy-prod.yml runs + failure rate
python -m podcast_obs cost-today                     # 24h LLM spend (Loki)
python -m podcast_obs logs --service pipeline --window 6h --contains OOM
python -m podcast_obs errors --window 24h            # Sentry issues
python -m podcast_obs alerts                          # Grafana alerts
python -m podcast_obs traces --limit 10              # recent Langfuse LLM traces
python -m podcast_obs correlate <run_id>            # EVERY signal for one run, joined

Every command prints a uniform JSON envelope — {ok, source, data | error, configured}. Exit code is 0 on success, 1 when the probe failed (unreachable / not configured), 2 on a config error. summary buckets sources into live / unconfigured / failed, so a local-only target still gives a useful glance (externals report unconfigured).

The logs command is the signal Sentry misses: it reads raw container logs from Loki (error-ish by default), including stderr tracebacks from pipeline subprocesses and ERROR/WARNING lines the Sentry SDK never wrapped.

MCP server (agent-facing)¶

Exposed as 27 tools (_build_tools), each taking an optional target. The two verb tools are the primary agentic surface:

obs_surface(surface=api|pipeline|player|operator) — observe one surface: RED metrics (VictoriaMetrics), errors (GlitchTip), error logs + traces (VictoriaLogs / VictoriaTraces), and — for the pipeline — the per-stage pipeline_stage rollup + LLM cost.
obs_investigate(trace_id|run_id|episode_id) — drill on one join key across every backend (span tree, logs, cost, errors, pipeline_stage).

Raw signals: obs_events (the emit_event stream — pipeline_stage / llm_cost / search_query in VictoriaLogs), obs_metrics (PromQL), obs_traces (VictoriaTraces spans). Plus the deploy probes (prod_health / prod_version / prod_resilience / prod_recent_runs / prod_recent_deploys / prod_usage / prod_cost_today / prod_recent_logs / prod_recent_errors / prod_recent_alerts / prod_recent_traces / prod_run_summary / prod_summary / prod_correlate) and the RFC-088 enrichment_* tools.

Backends (current stack). The Victoria* sources point at the self-hosted homelab box (VictoriaLogs / VictoriaMetrics / VictoriaTraces); the errors source points at GlitchTip via sentry_url. Configure with victorialogs_url / victoriametrics_url / victoriatraces_url / victoria_token / sentry_url (env PODCAST_OBS_* or a YAML victoria: / sentry.url block). The legacy Grafana-Cloud-Loki / Sentry-SaaS / Langfuse-Cloud sources remain available behind their own config for a legacy target.

Not read-only. enrichment_re_enable / enrichment_cancel mutate deploy state and are gated behind PODCAST_OBS_ALLOW_WRITES=1 (default off → they refuse). Everything else is read-only.

python -m podcast_obs serve --transport stdio                       # local agent
python -m podcast_obs serve --transport http --host 0.0.0.0 --port 8848   # networked control plane

Use stdio for a co-located agent; sse / http (streamable-http, default path /mcp) for a container other tailnet boxes can reach.

Docker (standalone control plane)¶

A light image (python:3.12-slim + httpx/PyYAML/mcp + the package — no pipeline deps) you run on your MBP, an Orb, or a Mac mini in the tailnet.

docker build -f docker/observability/Dockerfile -t podcast-obs:latest .

# one-shot probe
docker run --rm -e PODCAST_OBS_API_BASE=https://prod-podcast.<tailnet> podcast-obs summary

# the agent-facing control plane (MCP over http)
docker run --rm -p 8848:8848 \
  -v "$PWD/config/observability.yaml:/config/observability.yaml:ro" \
  -e PODCAST_OBS_CONFIG=/config/observability.yaml podcast-obs

See docker/observability/docker-compose.example.yml. To reach prod over Tailscale and be reachable by remote agents, run it on a tailnet host (host networking) or add a tailscale sidecar.

Task an agent to use it (with Grafana MCP and others)¶

Register podcast_obs as an MCP server in your agent. Claude Code, local stdio:

{
  "mcpServers": {
    "podcast-obs": {
      "command": "python",
      "args": ["-m", "podcast_obs", "serve", "--transport", "stdio"],
      "env": { "PODCAST_OBS_CONFIG": "/path/to/observability.yaml" }
    }
  }
}

Or a remote container over the tailnet:

{
  "mcpServers": {
    "podcast-obs": { "url": "http://mac-mini.tail-xxxxx.ts.net:8848/mcp" }
  }
}

Add the Grafana MCP server alongside it the same way. Division of labour:

podcast_obs — the fast cross-source glance: health, version, deploys, cost-today, raw error logs, Sentry issues, alerts. One call (prod_summary) tells you what's off.
Grafana MCP — deep drill-down: render a dashboard panel, query a specific metric over a range, inspect an alert rule.

Example you can hand an agent:

"Use podcast-obs prod_summary for prod. If cost-today is unusually high or there are recent error logs, pull the relevant Grafana dashboard via the Grafana MCP and summarise the spike; if a deploy failed, show its recent prod_recent_logs for the pipeline service."

The agent gets a cheap, structured first look from podcast_obs and escalates to Grafana only when something needs investigation — no operator ssh or dashboard-clicking.

Langfuse LLM tracing (#1052)¶

Langfuse is the AI-quality lens the cost/ops sources don't give: where cost-today answers "how much did we spend", Langfuse answers "what did each LLM call do" — a generation span per call (model / token usage in·out·total / cost / stage), grouped per run. (Per-call latency is a phase-2 item — the span is emitted at the post-call cost choke point, which carries no timing yet.) It coexists with the own solution (Loki llm_cost + corpus_manifest.cost_rollup + Sentry stay the source of truth for cost/ops); Langfuse is additive, not a replacement.

Two surfaces, one key pair (LANGFUSE_PUBLIC_KEY + LANGFUSE_SECRET_KEY; always set LANGFUSE_BASE_URL — these are the Langfuse SDK's own env names; unset silently defaults to Langfuse Cloud):

Pipeline emits — a hook at the provider cost choke point (record_provider_call_cost) emits one generation span per LLM call across all 8 providers. Enable-when-secret-present (Sentry pattern): a true no-op unless both keys are set, so dev / CI / offline runs stay silent. The SDK ships in [dev]; a runtime-only install adds the [langfuse] extra (pip install -e '.[langfuse]') — and the prod pipeline image bakes it. Langfuse is an optional o11y extension — see OBSERVABILITY_EXTENSIONS.md.
Control plane reads — the traces probe / prod_recent_traces MCP tool / Ops-view card query the same account back (Basic auth, httpx only — no SDK in the light control plane).

Hosting: point LANGFUSE_BASE_URL at the self-hosted homelab instance (http://homelab:4000) — never Langfuse Cloud. Leaving it unset falls back to Cloud (billed, off-tailnet), which is the one footgun here. Spans never block a run — every tracing entrypoint is wrapped, so a tracing failure is at most a missing span plus a debug log.

Correlation — one run, every signal (#1053)¶

The signals above are individually useful, but the real power is joining them. Every signal a run emits is stamped with the same run_id join key:

the Loki cost event carries run_id (and the run's log lines are prefixed [run=<id>] by the pipeline's CorrelationFormatter);
the Langfuse trace is seeded by run_id — its id is sha256(run_id)[:16], so the control plane addresses it directly, no search;
the Sentry scope is tagged run_id (and episode_id).

prod_correlate(run_id) / python -m podcast_obs correlate <run_id> fans out the run-scoped probes and returns them under one envelope:

{ run_id, live: [...], unconfigured: [...], signals: {
    trace:  { observations: [{stage, episode_id, model, cost, usage}, ...] },  # Langfuse
    cost:   { total_cost_usd, events: [...] },                                 # Loki
    errors: { projects: [...] },                                               # Sentry
    logs:   { lines: [...] } } }                                               # Loki

So an agent (or you) can take a run_id and answer "what did this run do, what did it cost per episode, and did it error?" in one call — each source degrading independently (configured=false) when its backend isn't wired. episode_id is set per episode during summarisation, so the trace's observations attribute cost to the right episode.

Validation¶

Unit tests: tests/unit/podcast_obs/ (config, every source, summary, MCP wiring).
Live E2E: tests/e2e_observability/ — asserts shape + invariants (not values, since live data changes) and self-skips when a source is unconfigured/unreachable. Run with real network:

.venv/bin/python -m pytest tests/e2e_observability/ -q --no-cov -p no:cacheprovider

GitHub runs live via your gh token; the rest light up as you provide their read-scoped tokens.

Prod observability control plane (podcast_obs, #803)¶