PRD-019 · Immersive Mode — WebXR AR (Android web + wrapped) + ARKit (iPhone wrapped)
Status · Draft v0.4 (all 4 v1 architectural decisions resolved 2026-05-16) Date · 2026-05-16 Owner · Marko Closes into · RFC-021 Slice gate · v1.x — last in the dependency chain (mobile + audio + sensory must ship first)
Why this is a PRD. Immersive Mode is the editorial peak of the product: solar system on your kitchen table, planets singing from their physical positions, a Sagan-register narrator explaining what you're seeing. It depends on the four already-shipped pairs (mobile wrapper, audio narration, sensory layer, image pipeline) and adds two new architectural commitments: a Three.js whole-codebase upgrade (r128 → current, touching all 7 existing 3D scenes), and a Capacitor + ARKit Swift plugin for iPhone parity (the only path to AR on iPhone, since Apple has not shipped WebXR). The PRD framing evolves PRD-015's "web is the product" stance — the browser product stays web-only; the mobile app product is allowed to harness native power where it unlocks things the web can't do (ARKit being the canonical example).
§why
The original orrery — the 18th-century brass instrument the product is named after — was a thing you held and turned with your hands. The whole product so far has been climbing back toward that tactile experience: 3D scenes you can orbit, planets that sing (PRD-017 sensory), a narrator that frames what you're looking at (PRD-016 audio). Immersive Mode is the moment it actually arrives — the solar system on a real table, in real space, with real spatial audio and a real voice in your ear.
This is the convergence feature. It does not introduce new content; it reframes everything already shipped through a different rendering target (AR camera passthrough), a different audio listener (XR camera), and a different default mode of attention (no UI chrome — narrator + planets + the room you're standing in).
It also unlocks a museum / classroom / projector deployment ("Exhibit Mode") — same codebase, no AR, but auto-narrated cinematic orbits suitable for an unattended large-screen install. One app, three use modes: phone-in-hand AR, kitchen-table AR, museum-projector cinema.
§audiences
| Audience | Why this lands the experience |
|---|---|
| Curious learner (at home) | Pull out their Android phone or wrapped Orrery on iPhone, tap "Enter AR", and walk around their solar system on the kitchen floor. Narrator guides them through. 10 minutes that they remember. |
| STEM student (classroom) | Teacher projects "Exhibit Mode" on a large screen; cinematic orbits + narrated tour run unattended. Students get phones and can scan a QR to take the same experience home. |
| Educator (museum / kiosk) | Exhibit Mode runs chrome-less on a kiosk monitor + speakers; visitors stop, listen, and a QR offers AR continuation on their own device. |
| Enthusiast (cinematic) | Wants the editorial peak — pale blue dot moment, Cernan's last words, the Kepler chord — delivered in the most immersive form available. AR + narrator + sonification + sensory haptics all firing. |
Note: iPhone Safari users do not get AR. They get the rest of the product (3D scenes + sensory + narration). AR is exclusive to (a) any Android Chrome (web or wrapped), and (b) iPhone in the wrapped Capacitor app via the ARKit Swift plugin.
§what's already shipped (immersive-readiness inventory)
| Capability | Status | Source |
|---|---|---|
| 11 primary nav routes (7 with 3D scenes) | shipped (v0.6) | TA.md |
| Capacitor mobile wrapper (Android-first, iPhone follows) | planned (PRD-015 / RFC-018, v0.8) | — |
| Audio narration episode system (PRD-016 Curator + Guide + Enthusiast) | planned (v0.9) | RFC-019 |
| Sensory layer — gyro + sonification + haptics + audio-bus ducking | planned (v1.x) | PRD-017 / RFC-020 |
| Image Pipeline v2 — vision scoring + smart-crop variants | planned (v1.x) | PRD-018 / RFC-022 |
| Three.js r128 across 7 existing 3D scenes | shipped (v0.6) | TA.md §3D |
ADR-057 (no localStorage; only orrery_locale cookie persists) | shipped | ADR-057 |
What this means: Immersive Mode lands AFTER the entire stack above. It is last in the dependency chain. v1.x ship gate.
§architectural axis — web vs mobile-app split
Evolves PRD-015's framing. Made explicit because this is the first feature where the split actually matters:
| Surface | What it is | What it CAN do | What it CANNOT do |
|---|---|---|---|
Web browser (chipi.github.io) | The web product. Mobile + desktop browser. | Web standards only. Three.js, Web Audio, Vibration API on Android, WebXR on Android Chrome. | No native APIs. iPhone Safari: no AR (WebKit lock). |
| Capacitor mobile app (App Store + Play) | The wrapped app. Web shell + native plugins. | Everything the browser does, PLUS native plugins where they unlock real capability. Capacitor Haptics (iOS Taptic). ARKit (iPhone AR) via Swift plugin. Capacitor's deep-link / share / app-state APIs. | The same web bundle as the browser product. The native layer is thin and specific to capabilities the web can't do. |
Where iPhone gets AR: the wrapped app only. iPhone Safari users get the rest of the product (3D + sensory + narration). The wrapped app is the path to real AR on iPhone.
Where Android gets AR: everywhere. Android Chrome (web) and Android wrapped both run the same WebXR code path.
§goal
Ship Immersive Mode on:
- Android web (Chrome) → WebXR AR on four globe scenes
- Android wrapped (Capacitor) → same WebXR AR
- iPhone wrapped (Capacitor + ARKit Swift plugin) → ARKit AR on four globe scenes
- iPhone Safari → no AR (sensory + narration only — already covered by PRD-017 + PRD-016)
- Desktop browser → "Exhibit Mode" (cinematic auto-narrated orbits, no AR, chrome-less)
Four globe scenes: /explore (solar system), /earth, /moon, /mars. No /fly mission arc in AR for v1 — AU-scale on a tabletop is a hard design problem; deferred to v2. No /iss / /tiangong station-tabletop AR — defer to v2.
Vision Pro: dropped from v1 scope entirely. Marko's call 2026-05-16. The 4 mobile platforms (Android web + Android wrapped + iPhone wrapped + iPhone Safari fallback) are the focus.
§user stories
US-1 — Kitchen-table solar system. User on Android Chrome opens Orrery, navigates to /explore, taps the "Enter AR" button in the nav. Browser requests camera permission. After grant, user taps the floor or table to place the solar system. Saturn appears at scale on the surface; the planets begin their orbits. The Guide voice (auto-played 2 s after placement, per PRD-016) starts narrating. Spatial audio from each planet's PannerNode follows their physical position on the table. User walks around — planets stay locked to the table; voice stays in centre of their head.
US-2 — Same experience on iPhone in the wrapped app. User opens Orrery via the App Store (PRD-015's wrapped app). Same "Enter AR" button. The ARKit Swift plugin activates instead of WebXR; the plugin exposes the same JS API surface so the Three.js code path doesn't change. User experience identical to Android.
US-3 — Exhibit Mode (museum / classroom projector). Operator opens /?mode=exhibit on a desktop browser, full-screens. The page hides all chrome (nav, footer, HUD), starts a 90-minute cinematic auto-narrated orbit sequence through /explore → /earth → /moon → /mars, ending on a Curator close from PRD-016's Full Tour. A QR code in the corner offers AR continuation on visitors' devices.
US-4 — iPhone Safari graceful fallback. User on iPhone Safari opens Orrery, sees the same nav, but the "Enter AR" button is greyed out with a tooltip: "AR requires the Orrery app on iOS — get it from the App Store [link]." The rest of the product works fully (3D + sensory + narration).
US-5 — Narrator drives the experience. Within 2 seconds of AR placement, the Guide voice begins the screen's episode. Sonification ducks per PRD-017 §M9 audio-bus contract (−34 dB). User listens; planets sing under the voice. Episode ends → sonification restores to full level. User can pause / skip / restart via overlay.
§must-have requirements
| ID | Requirement |
|---|---|
| M1 | Single "Enter AR" button in the nav on the 4 globe routes (/explore, /earth, /moon, /mars). Visibility per platform: Android (any browser) shows; iPhone wrapped app shows; iPhone Safari shows greyed-out with tooltip + App Store link; desktop hides. |
| M2 | Android (web + wrapped): WebXR AR via Three.js's WebXRManager. Requires Three.js WHOLE-CODEBASE UPGRADE (see M14) to a version with reliable hit-test support. Uses ARCore via the browser; no Android-side native plugin. |
| M3 | iPhone wrapped: ARKit AR via a new Capacitor Swift plugin (@orrery/ar-bridge). The plugin wraps ARKit's ARSession + ARHitTestResult + ARAnchor APIs and exposes them with the SAME JS API surface as WebXR (so the Three.js code path is shared). |
| M4 | AR scope = 4 globe scenes for v1: /explore, /earth, /moon, /mars. /fly (mission arc), /iss, /tiangong deferred to v2. |
| M5 | Spatial audio. When AR session starts, the Three.js AudioListener swaps from the flat-screen camera to the XR camera. PRD-017 sonification graphs are reused verbatim — only the listener source changes. Each planet's PannerNode tracks its 3D world-space position; user walks around and hears Saturn from where Saturn physically is on the table. |
| M6 | Narrator auto-play (NE-B). 2 seconds after AR placement is confirmed (user has tapped the table), the screen's Guide episode begins. Narrator is centre-panned and spatially fixed 30 cm above and behind the listener's head (omniscient narrator position — N-B). Sonification ducks to −34 dB per PRD-017 M9. |
| M7 | Headphone-aware audio rendering. navigator.mediaDevices.enumerateDevices() detects connected audio output. Headphones → HRTF spatial audio (PannerNode.panningModel = 'HRTF'). Speakers → equal-power stereo. Auto-switch on plug/unplug events. |
| M8 | Performance. 72 fps target on Pixel 6a (representative mid-tier Android), iPhone 12 (representative iPhone), both in AR mode. Simplified starfield in AR (~100 stars vs 2000 in flat-screen). No particles / comets / orbit trails in AR. |
| M9 | Bundle isolation. XR module (Three.js XR scene builders + AR control overlay) is a Vite lazy chunk, loaded only when user taps "Enter AR". <50 KB minified gzipped. Exhibit Mode is a separate lazy chunk, <20 KB. Zero impact on the flat-screen bundle. |
| M10 | AR session storage = in-memory only. Per ADR-057, no localStorage. AR placement state (anchor position, narrator playback position) is runtime-only; exiting the AR session resets state. NO sessionStorage either — the original draft proposed sessionStorage; replaced with in-memory state per CLAUDE.md / ADR-057. |
| M11 | Exhibit Mode (M-B chrome-less). ?mode=exhibit URL param. Hides all chrome (nav, footer, HUD overlays). Auto-starts a cinematic playlist: 22 min /explore (Curator + Kepler chord + planet tour) → 18 min /earth (Earth orbit regime overview) → 22 min /moon (terminator + far-side + Apollo sites) → 18 min /mars (rover sites + signal-delay moment) → 10 min Curator close. ~90 min total. Loops. QR code static-corner overlay pointing to the AR experience on visitor's device. |
| M12 | iPhone Safari fallback messaging. "Enter AR" button on iPhone Safari is visually distinct (greyed-out + small "iOS app" badge). Tap opens a non-modal sheet: "Real AR on iPhone needs the Orrery app. [Download from App Store]" + "Or continue with the sensory experience (tilt + sound + haptics)." Honest framing; not a dead-end. |
| M13 | @capacitor/haptics integration in AR. AR-specific haptic events (per RFC-021 §haptics-in-AR): anchor placement confirmed → light pulse; narrator-section transition → light pulse; episode end → triple light pulse. Reuses the patterns already specified in PRD-017 RFC-020 §5.2. |
| M14 | Three.js WHOLE-CODEBASE UPGRADE — v1 prerequisite. All 7 existing 3D scenes migrate from r128 to current Three.js (likely r170+) before Immersive Mode work begins. This is a meaningful undertaking — see §dependencies for scope estimate. Justified by: (a) WebXR hit-test reliability requires modern Three.js, (b) keeping two Three.js versions in the build is a maintenance trap. |
| M15 | ARKit Swift plugin scope. @orrery/ar-bridge Capacitor plugin (Swift package) exposes 6 methods: requestSession() / endSession() / hitTest(x, y) / addAnchor(anchor) / removeAnchor(anchor) / getCameraPose(). Plus 3 events emitted to JS: session-started, frame (per RAF with camera pose), session-ended. ~600 lines Swift. Lives in the Capacitor ios/ directory; no separate iOS repo. |
| M16 | AR camera passthrough video respects user privacy. Camera frame data is NEVER stored, transmitted, or logged. Only spatial-tracking data (camera pose + anchor positions) is used by JS. Documented in the privacy policy + the AR permission grant UI ("Orrery uses your camera to place the solar system in your room. Camera frames are processed on your device and never sent anywhere."). |
§should-have requirements
| ID | Requirement |
|---|---|
| S1 | Single-tap "Re-place" gesture during an AR session: user taps the table again to re-anchor the scene. |
| S2 | Visual on-boarding overlay first time the user enters AR: 3-second hint showing the "tap surface to place" gesture, then dismisses on tap. Once per session (in-memory; no persistence). |
| S3 | "Exit AR" button visible in the AR session UI (top-right corner of overlay), 44×44 px. |
| S4 | Exhibit Mode's QR code link is configurable per deployment (URL param ?qr=<shortlink>) — museum operators can point it to their landing page. |
| S5 | Per-platform telemetry in the analytics pipeline (PRD-015 already wires the global click delegation): ar-session-start / ar-session-end / ar-error events with platform context ('android-web' / 'android-wrapped' / 'iphone-wrapped'). Helps prioritise v2 work. |
§will-not-have (v1)
- Vision Pro support. Dropped from v1 scope (Marko 2026-05-16). Focus is mobile (Android + iPhone).
/flymission arc in AR. AU-scale on a tabletop is a hard design problem; deferred to v2./iss//tiangongstation-tabletop AR. Defer to v2.- AR on iPhone Safari. Apple has not shipped WebXR. Honest framing in M12 fallback.
- AR occlusion (hands blocking planets). Requires ARCore Depth API + ARKit Scene Geometry — both v2.
- Anchor persistence across sessions. Put phone down, pick up, anchor still there. Requires cloud anchors or persisted ARCore state; v2.
- Multi-user shared AR. Two phones, same scene. Requires Google Cloud Anchors or ARKit collaborative sessions; v2.
- Script-aware narrator-sonification semantic ducking. v1 uses amplitude-based ducking only (PRD-017 M9). Semantic tags ("duck during equation explanation but not during transition") deferred to v1.x.
- Exhibit Mode synced multi-screen (multi-projector wall). WebSocket-coordinated sync between 2+ projectors. v2.
- Vision Pro gaze + pinch UX adaptation. Dropped along with Vision Pro scope.
§success-criteria
Editorial:
- A first-time AR user on Android Chrome at
/exploretaps "Enter AR", places the solar system, and within 2 seconds the Guide narration starts. The first 10 minutes are uninterrupted (no UX breakdowns; no 3D drift; no audio glitch). - iPhone wrapped app delivers IDENTICAL editorial experience to Android Chrome (per US-2). Cross-platform user study (Marko + 2 reviewers, one Android one iPhone) confirms parity.
- Exhibit Mode runs unattended for 8+ hours without crash on a desktop browser (manual test).
Technical: 4. 72 fps held on Pixel 6a + iPhone 12 in AR mode for full 90 s sessions (Lighthouse mobile + Chrome DevTools / Xcode Instruments profiling). 5. XR + Exhibit chunks total < 70 KB minified gzipped; zero impact on flat-screen bundle size. 6. Three.js whole-codebase upgrade lands with zero visual regression on the 7 existing 3D scenes (validated via the visual regression baselines from tests/e2e/visual.spec.ts + manual review of each scene). 7. ARKit Swift plugin builds clean on Xcode 16+; no warnings; passes App Store review.
Operational: 8. iPhone Safari users see the M12 fallback messaging on first AR-button tap (no broken-button confusion). 9. Camera permission request UX is honest about what data is used (M16).
§dependencies
Must ship first (entire stack):
- PRD-015 / RFC-018 — Capacitor mobile wrapper (Android-first; iPhone follows). Without it, no wrapped app exists.
- PRD-016 / RFC-019 — Audio narration. Without it, no Guide voice to auto-play.
- PRD-017 / RFC-020 — Sensory layer. Provides the sonification graph that Immersive Mode reuses + audio-bus ducking contract.
- PRD-018 / RFC-022 — Image Pipeline v2 (loose dependency — not strictly required, but the smart-crop variants improve AR texture quality).
Three.js upgrade — v1 prerequisite (NOT a v1 feature):
- All 7 existing 3D scenes (
/explore,/flyheliocentric,/flycislunar,/earth,/moon,/mars,/iss,/tiangong) migrate from r128 to current Three.js (likely r170+). - Estimated 4-6 weeks engineering work (audit breaking changes → scene-by-scene migration → visual regression sweep → fix-up).
- Risk: regressions across the existing 3D corpus. Mitigation: aggressive use of
tests/e2e/visual.spec.tsbaselines + the audit-report pattern from PRD-018.
ARKit Swift plugin — v1 prerequisite:
- New
@orrery/ar-bridgeCapacitor plugin, ~600 lines Swift. - ~3-4 weeks Swift + Capacitor plugin development.
- Owned by Marko (per PRD-015 §iOS code-signing — Marko owns the iOS path).
ADR-057: Strict — no localStorage, no sessionStorage. AR session state is in-memory only.
§resolved decisions
Resolved 2026-05-16 in conversation with Marko.
- Three.js — RESOLVED: Upgrade across the whole codebase (NOT just the XR module). All 7 existing 3D scenes migrate from r128 to current. Single Three.js version in the build. v1 prerequisite, not a v1 deliverable.
- Narrator — RESOLVED: Bake in from v1. Immersive Mode ships only after PRD-016 audio lands (v0.9). Guide episode auto-plays 2 s after AR placement.
- AR scope — RESOLVED: 4 globe scenes only (
/explore,/earth,/moon,/mars)./flymission arc deferred to v2;/iss+/tiangongdeferred. - iPhone AR path — RESOLVED: Path A — Capacitor + ARKit Swift plugin (
@orrery/ar-bridge). The only way to ship real AR on iPhone since Apple has not shipped WebXR. - Web-vs-app split — RESOLVED: Web product stays web-only; mobile app is allowed native power. "Browser product" is web-pure; "mobile app product" can use Capacitor plugins (including ARKit) where they unlock things the web can't. Evolves PRD-015's framing; codified in §architectural-axis.
- Vision Pro — RESOLVED: Dropped from v1 entirely. Focus on mobile (Android + iPhone). Vision Pro is not a v1 audience.
- AR session storage — RESOLVED: In-memory only. Original draft proposed sessionStorage; corrected per ADR-057 to runtime-only state.
- Headphone-aware audio rendering — RESOLVED: Auto-detect via
navigator.mediaDevices.enumerateDevices()→ HRTF on headphones, equal-power stereo on speakers (M7).
§open questions
- ARKit Swift plugin maintenance burden. ~600 lines of Swift + Apple's ARKit API evolution. Who owns it long-term? Marko (per PRD-015 iOS signing) is the default — confirm acceptable maintenance commitment.
- Three.js upgrade risk envelope. Estimated 4-6 weeks. Open: how aggressively to validate? Visual regression baselines (tests/e2e/visual.spec.ts) cover ~6 surfaces; the 7 3D scenes are not in that suite (canvas regressions are flaky). Manual review pass required. Operational, not architectural.
- AR onboarding UX iteration. S2 first-time hint may need 2-3 design iterations before launch. Implementation-time.
- Exhibit Mode QR code → AR continuation handshake. Operator deploys Exhibit Mode on a kiosk; visitor scans QR; the visitor's phone opens the AR experience deep-linked to the current scene. Implementation:
orrery://?audio=...&mode=ar&scene=exploredeep-link (per RFC-018 §7). Refine at impl time. - iPhone wrapped app — TestFlight vs direct App Store path. Marko's Apple Dev account (PRD-015) handles both. Open: ship to TestFlight first for 2 weeks before public App Store? Recommend yes.
PRD-019 · Orrery · Immersive Mode · Drafted 2026-05-16 · Closes-into-RFC-021