ADR-082 — Telemetry environment ladder (dev → staging → prod, isolated projects)

Status · Accepted Date · 2026-07-24 Related ADRs · ADR-067 (Sentry/GlitchTip error tracking — extended here), ADR-081 (self-hosted Umami analytics — extended here), ADR-070 / ADR-071 (docs vs app deploy split, the staging workflow this retargets) Supersedes wording in · ADR-067 §"Local-dev posture" and ADR-081 §"Why env-var-gated" (both said vite dev is hard-silent — no longer true; see below)

Context

Two sibling ADRs already gate browser telemetry on env vars, not hostname: ADR-067 (Sentry SDK → self-hosted GlitchTip) and ADR-081 (self-hosted Umami). Both shipped with a binary posture: a deploy bakes the PUBLIC_* vars → telemetry is live; everything else (vite dev, vite preview, CI, forks) is silent.

That binary posture had two gaps once the product went to production on orrerylearn.com (VPS + Cloudflare) while the chipi.github.io/orrery gh-pages mirror kept deploying:

1. The gh-pages deploy is not a "preview" — it is a real, publicly reachable environment. It was labelled production in its Sentry env tag and set no Umami at all, so its error traffic polluted the prod GlitchTip project and its usage was invisible. A shared prod dashboard that mixes two live environments cannot answer "is this error happening to real users?"
2. Local vite dev was telemetry-blind. A crash or a UX dead-end an engineer hit while developing produced nothing — no error in any GlitchTip project, no event in any Umami site. The one environment where we iterate fastest was the one with no feedback loop.

Decision

Model telemetry as a three-rung environment ladder. Each rung reports to its own isolated GlitchTip project + Umami site, tagged by environment, so no rung's data ever contaminates another's.

Rung	Where	GlitchTip project	Umami site	environment tag	How it's selected
prod	orrerylearn.com (VPS + CF)	18 (was recorded as 4, which never existed; corrected 2026-10-17)	prod site (4a25d8da…)	production	deploy-prod.yml bakes PUBLIC_SENTRY_DSN + PUBLIC_UMAMI_* from the prod GH environment
staging	chipi.github.io/orrery (gh-pages)	6	staging site (6e7ddfce…)	staging	staging.yml bakes them from the new staging GH environment
dev	maintainer's laptop, vite dev	30 (orrery-dev; corrected 2026-10-17, 7 never existed)	"Orrery (dev)" site (90cbabf7…; created 2026-10-17)	dev	fallback constants in sentry.ts / analytics.ts, reached only over the tailnet TLS nodes (glitchtip. / umami.tail6d0ed4.ts.net)

Concretely:

• preview.yml → staging.yml. The gh-pages deploy is renamed and declares environment: staging (a new GH environment holding the staging-only DSN + Umami id), mirroring deploy-prod.yml's environment: prod. Its Sentry env tag is staging, not the old mislabeled production. (References updated in release.yml, deploy-docs.yml, docker-e2e.yml.)
• Dev rung. In vite dev with no deploy-injected PUBLIC_* override, sentry.ts falls back to DEV_SENTRY_DSN (GlitchTip project 30 orrery-dev; the original project 7 never existed, so dev events were dropped until 2026-10-16) and analytics.ts falls back to DEV_UMAMI_HOST/DEV_UMAMI_WEBSITE_ID (the dev Umami site). Both endpoints resolve only via the Tailscale MagicDNS host homelab (GlitchTip :8090, Umami :3001) — no fixed IP.
• Worktree tag. vite.config.ts injects the current git branch as the __DEV_WORKTREE__ define; sentry.ts adds it as a worktree tag on dev events only, so errors from parallel local worktrees/agents stay distinguishable in the dev project. It is never attached to staging/prod events.

Why this preserves fork-silence (the load-bearing property)

The binary posture's one real guarantee — a stranger who clones the repo sends nothing — is kept, now by construction rather than by a dev hard-block:

• Deploy rungs (prod/staging): the PUBLIC_* values live in GH environment secrets a fork doesn't have. A fork's build bakes empty → SDK/tracker no-op.
• Dev rung: the fallback DSN/host is the Tailscale-only name homelab. Only a device on our tailnet resolves it. On a stranger's machine vite dev resolves nothing, the transport fails, and no event leaves the browser. The DSN key + Umami id are public browser ids (they ship in the bundle by design — same as any Sentry DSN), so committing them leaks nothing.

So the guarantee moved from "dev is hard-blocked" to "dev's endpoint is unreachable off the tailnet." Same outcome for forks; a working feedback loop for the maintainer.

Consequences

• vite dev is no longer silent for the maintainer. Any doc that says "local dev is silent / dev sends nothing" is now only true for non-tailnet machines. Updated: README §Privacy, src/lib/analytics.ts module JSDoc, docs/guides/observability.md, ADR-067 / ADR-081 (amendment pointers to here).
• New infra (homelab, owned by the infra repo): GlitchTip project 6 (orrery-staging) + 30 (orrery-dev); Umami sites 6e7ddfce… (staging) + 90cbabf7… (dev, created 2026-10-17); the GH staging environment + its secrets.
• vite preview, screenshot pipeline, e2e stay silent — they are not dev and carry no baked PUBLIC_*, so no rung resolves. The public prod dataset reflects real visitors only.
• Reading a dev error requires being on the tailnet. Acceptable — dev telemetry is a maintainer convenience, not a product surface.

Not covered / out of scope

• Native mobile crash reporting (@sentry/capacitor, #428) plugs into the same GlitchTip. Originally scoped out of the browser ladder; the 2026-07-24 amendment below folds mobile INTO the ladder as a platform segment.
• Log shipping (ADR-068 Grafana Agent / VictoriaLogs) is a server-side concern, not browser telemetry, and is untouched here.
• The public-vhost hardening that hides the GlitchTip/Umami admin surfaces is the infra repo's responsibility (as in ADR-081).

Amendment — 2026-07-24 (mobile is a segment of the ladder, not a separate app)

#428 initially gave the Capacitor shells their own environment value (mobile-<platform>), which overloaded environment with the platform and so collided with using it for the tier — a native crash always read mobile-ios regardless of whether it came from a simulator or an App Store build. Decision: mobile is a segment of the same application, not a separate app, and rides the same tier ladder. Two independent dimensions:

Dimension	Values	Meaning
environment	dev / staging / prod	the tier — uniform for web AND native
platform tag	web / ios / android	the segment — splits app from browser

• sentry.ts: environment now resolves the same way for web and native (PUBLIC_SENTRY_ENVIRONMENT → vite dev→dev → else prod); the mobile-<platform> override is gone. Native still inits via @sentry/capacitor (+ the @sentry/svelte sibling) and carries the platform tag. Result: prod project 18 holds {web, ios, android} events all environment=prod, filterable by platform; staging project 6 likewise. Covered by sentry.native.test.ts.
• Mobile build picks the tier by what it bakes (native builds are local, not CI). Two recipes — see docs/guides/mobile-build-and-deploy.md §Ship:
  • Simulator / staging: export the staging DSN + PUBLIC_SENTRY_ENVIRONMENT=staging + the staging Umami id, then sync:mobile → crashes land in staging project 6, pageviews in the staging Umami site.
  • Release / App Store: export the prod DSN + =prod + the prod Umami id → prod project 18. "What I bake at the end always goes to prod."
• Umami is identical per tier, device-agnostic (operator decision 2026-07-24). One Umami site per tier; no platform dimension — Umami's own UA-based device breakdown already separates desktop / mobile. The app's capacitor://localhost origin is accepted (the sites don't restrict domain — verified by test ingest). The only requirement: the mobile build must bake PUBLIC_UMAMI_* (per tier) or the in-app analytics is silent — #428's recipe baked only the Sentry DSN.
• Orthogonal to STREAM_ORIGIN (the asset host): a build's telemetry tier and its asset origin are set independently at build time. ADR-083 then couples them behind a single runtime switch for internal mobile builds (staging↔prod for telemetry + assets together), while App Store release stays prod-locked.

Amendment — 2026-10-17 (isolation audit: ids and the dev rung's targets)

An end-to-end audit (operator: "staging and dev must never drip into prod"), with the homelab session reading GlitchTip, Umami and VictoriaLogs/Traces:

• Project ids. prod is GlitchTip 18 (orrery-prod); there was never a project 4. staging is 6 (slug orrery), dev is 30 (orrery-dev), and the servers report to 31 (orrery-lab-api) and 32 (orrery-mcp). The Decision table and the docs that copied "4" are corrected.
• Isolation holds today. Live prod env.js → 18 + prod Umami site; staging → 6 + staging site. CI/e2e builds bake no target, and CI runners can't reach the tailnet telemetry hosts. All orrery logs/traces come from the VPS, tagged environment=prod.
• The dev rung reported nothing. Its fallbacks pointed at homelab:8090 and homelab:3001, which the mini binds to loopback only, and the dev Umami site 1d2f214c… doesn't exist. A vite dev probe error never reached project 30. The dev DSN host becomes glitchtip.tail6d0ed4.ts.net (the key is project 30's), and dev analytics go to a new "Orrery (dev)" Umami site 90cbabf7… on umami.tail6d0ed4.ts.net (operator: "it's kind of cool to follow what people are developing").
• Servers join the ladder. server/observability resolves the tier from ORRERY_ENVIRONMENT (else NODE_ENV: production → prod, test → test, anything else → dev). A deploy value always wins. dev and staging have built-in targets (TIER_TARGETS); prod has none in code, so a prod run missing a target sends nothing for it, and tests never send. dev: errors → project 30 (tag service), traces → vtraces with deployment.environment=dev, logs pushed straight to VictoriaLogs (no Alloy on a laptop), Umami → the dev site. staging (defined for a future staging server): errors → project 6, traces → vtraces, Umami → the staging site; logs via Alloy like prod.
• Past drip. The prod Umami site holds 183 pageviews with hostname localhost (2026-08-19 → 09-10, none since). Cleanup is the operator's call.
• validate-prod now carries the analytics opt-out cookie, so a validation run is never a prod visitor.