ADR-094 — Server-side observability for lab-api and mcp (errors, traces, metrics, access logs, analytics, LLM)

Status · Accepted · 2026-10-16 · Operator-directed ("add all o11y elements to lab-api and mcp") · amends the OTLP non-goal in RFC-025 and the errors-only scope of ADR-067 for the two server processes only (the browser stays errors-only) · builds on ADR-081 (Umami) and ADR-082 (telemetry env ladder)

Context

RFC-025 ruled out OTLP traces because Orrery had "no traceable workload (static SPA, no server)". It said to revisit "when the VPS adds… a daemon". It now has two: orrery-lab-api (OAuth AS + /ask over the LiteLLM gateway) and orrery-mcp (64 physics tools over MCP). On 2026-10-16 their whole signal was one startup log line each:

• no error tracking (there was no server-side Sentry);
• no traces (VictoriaTraces had no orrery-* services);
• no request metrics;
• no per-request log line;
• no record of which MCP tools are used;
• /ask's LLM calls reached Langfuse with nothing to join them to a request.

The podcast services on the same VPS already ship all six, and the homelab stack (VictoriaLogs/Metrics/Traces, GlitchTip, Umami, Langfuse) is the shared sink.

Decision

Both servers get the same six signals, modelled on podcast's player-api / player-mcp, all off unless configured. With every variable unset, nothing initialises and nothing leaves the process. That keeps tests, CI image smoke tests and forks silent.

1. Errors → GlitchTip via @sentry/node. Projects 31 orrery-lab-api and 32 orrery-mcp; DSNs are deploy secrets. Errors only: tracesSampleRate unset and Sentry's own OTel setup skipped, so it doesn't fight the tracer in (2). environment = prod, release = sha-<7> (homelab ADR-0010; the browser's gha-<8> is left as is). PII: no user email, no request bodies, no prompt text.
2. Traces → VictoriaTraces, OTLP http/protobuf to OTEL_EXPORTER_OTLP_TRACES_ENDPOINT. The tracer is set up manually: the servers ship as one esbuild bundle with no node_modules, so require-hook auto-instrumentation is not available.
   • Resource: service.name = orrery-lab-api | orrery-mcp (the same value as the logs' service label), deployment.environment = prod, service.version = sha-<7>.
   • Spans: one server span per request (inbound traceparent honoured), one span per LLM call, one span per MCP / /ask tool call.
   • /health and /metrics are never traced.
3. Metrics → VictoriaMetrics via prom-client, served on a separate internal port (METRICS_PORT, not published to the host, so never reachable through Caddy).
   • Request count and latency by route and status.
   • For mcp, mcp_tool_calls_total{tool,outcome} and mcp_tool_duration_seconds{tool}; tool is bounded by the registry.
   • Scraped by the VPS Alloy from orrery.alloy, using the Docker-discovery → bridge-IP pattern base.alloy uses for the podcast APIs.
4. Access logs: one line per request: METHOD route -> status in N.Nms trace_id=<32hex>. Alloy's existing trace-id stage turns it into a link from a log line to its trace. /health and /metrics are not logged.
5. Analytics → Umami: mcp sends one server-side event per tool call to the Orrery site (hostname mcp.orrerylearn.com) with {tool, outcome}. It never carries user ids, arguments or prompt text.
6. LLM → Langfuse via LiteLLM metadata: /ask sends metadata with:
   • trace_id = the OTel trace id;
   • trace_name orrery ask;
   • session_id = a per-request id (not a user id);
   • generation_name = the tool round;
   • trace_release = sha-<7>.

Consequences

• New runtime dependencies: @sentry/node (pinned to the same version as the rest of @sentry/*, which @sentry/capacitor enforces), @opentelemetry/{api,sdk-trace-base,exporter-trace-otlp-proto,resources,semantic-conventions,core} (on the line already installed), and prom-client.
• Deploy: new GH prod secrets for the two DSNs. The OTLP endpoint and Umami site id are non-secret. Each value has to be added in three places (secret env, .env printf, compose environment:).
• Scrape order: deploy the images first, then the orrery.alloy scrape, or up reads 0 until the new images serve /metrics. This is the podcast lesson recorded in base.alloy.

Alternatives considered

• Auto-instrumentation (@opentelemetry/auto-instrumentations-node): needs unbundled node_modules and require hooks. Rejected for the single-file images.
• Sentry tracing instead of OTel: it would send traces to GlitchTip, not VictoriaTraces, so they wouldn't join with logs and metrics.
• /metrics on the public port, blocked at Caddy: one Caddyfile mistake away from public. Rejected for the separate internal port.