RFC-020 · Sensory Layer — gyroscope, sonification, haptics
Status: Draft v0.4 · 2026-05-16 (all v1 architectural decisions resolved) · Closes: PRD-017
Why this is an RFC. The architecture binds every 3D scene's camera controller (gyro coexistence with touch via
pointerdown/pointerup), the audio output channel (sonification ducking when PRD-016 narration plays), the haptics path (webnavigator.vibratevs Capacitor@capacitor/haptics), and the per-route sonification designs that take editorial work to author. These are interlocking decisions; one wrong cut early forces ugly retrofits later, especially the audio-channel coexistence with the narration system shipped one slice earlier (PRD-016 / RFC-019).
1 · Architecture overview
┌───────────────────────────────┐
│ Master sensory toggle (nav) │ default OFF
│ 44×44 px · waveform+compass │
└──────┬────────────────────────┘
│ tap → enable; long-press → settings sheet
▼
┌───────────────────────────────┐
│ Sub-toggles │
│ GYRO (mobile, 3D routes) │ → DeviceOrientation API + camera
│ AUDIO (web + Capacitor) │ → Web Audio API per-route graphs
│ HAPTIC (Android web; both via Capacitor) │ → Vibration / Capacitor Haptics
└──────┬────────────────────────┘
│
├─ GYRO → reference-frame calibrated; touch pauses
├─ AUDIO → per-route oscillator graph; ducks under narration
└─ HAPTIC → discrete events from physics thresholdsState: in-memory only (per ADR-057). Reload resets to OFF.
2 · Six anchor design decisions (locked from v0.1 draft, carried forward)
These were debated and chosen in the original RFC-018 draft (now renumbered). All six stand for v0.3:
| ID | Choice | Reason |
|---|---|---|
| G-C | Gyro mapping = reference-frame calibrated | Toggle-on captures device orientation as "home"; subsequent rotations are deltas. Avoids absolute-angle drift; matches how users intuitively expect a "tilt to look around" affordance to work. |
| T-B | Gyro + touch = pause-on-touch | Active touch suppresses gyro input; 200 ms after touch-end, gyro resumes from the new position. No snap-back. Lets users use either input naturally without conflict. |
| S-C | Sonification = per-screen specialisation | Each route gets its own sonic identity (Kepler chord, mission-arc 4-stream, regime drones, agency tones). Not a single universal sound. More authoring work; pays back in editorial weight. |
| I-B | Modality interaction = haptics confirm audio events | When a haptic pulse fires, an audio event also plays (and vice versa for the major events). The two senses confirm each other; reduces ambiguity on whether the user's input "took". |
| U-2 | Toggle UX = master + long-press settings sheet | Single 44 px toggle in nav; long-press opens a sheet with three sub-toggles. Surfaces simple "on/off" for most users; gives power users granular control. |
| P-A | iOS permission = on toggle tap | DeviceOrientationEvent.requestPermission() fires when the user taps the master toggle for the first time. Never on page load. Honours iOS's "permission requires user gesture" rule. |
3 · Sonification — per-route designs (all 11 routes)
Per-route specialisation (S-C). Original 6 routes designs preserved verbatim from v0.1; 5 new route designs added for v0.3 to match Marko's "all 11 routes" answer.
3.1 · Reference frequencies + scaling
BASE_FREQ = 220 Hz (A3 — Earth's reference oscillator on /explore)
SCALE_EXP = 0.5 (square-root scaling of planetary angular velocity → frequency)
DUCK_GAIN = 0.02 (≈ −34 dB; sonification level while narration plays)
ATTACK_MS = 50 (event envelope on selection / state change)
RELEASE_MS = 150 (envelope release; matches the visual transition)3.2 · Original 6 routes (carried forward unchanged)
/explore — Kepler chord
8 oscillators, one per planet. Frequency = BASE_FREQ × (ω_planet / ω_earth)^SCALE_EXP. Sine waveforms for inner planets (Mercury–Mars), triangle for outer (Jupiter–Neptune). Each planet's PannerNode tracks its 3D position; listener orientation tracks camera. Selected planet boosts its oscillator gain by +6 dB; others duck −3 dB.
/fly heliocentric — mission narrative (4 streams)
- Velocity → pitch (sine, 110–880 Hz log-mapped, dominant stream)
- Distance from Earth → reverb depth (
ConvolverNodewet/dry mix 0–0.7) - Fuel remaining → harmonic distortion (
WaveShaperNode, more distortion as fuel drains) - Signal delay → echo (DelayNode, delay time = light-time in seconds; CAPCOM-feel)
- Arrival event = major chord (root + major third + perfect fifth, 200 ms envelope)
/fly cislunar — same 4 streams adapted
Distance → reverb is muted (cislunar distances are short); signal-delay echo is exaggerated (Earth-Moon round-trip ~2.6 s is the editorial point). All other streams as /fly heliocentric.
/fly porkchop magnifier
Cell-tap = single tone, pitch maps to Δv (lower Δv = lower pitch — "easier missions sound deeper"). Drag = continuous sonification of the cells crossed. Solver-complete = three-note chord (200 ms attack envelope per note).
/earth — orbital regime drones
| Regime | Waveform | Freq | Notes |
|---|---|---|---|
| LEO | Sine | 110 Hz | Soft, continuous |
| MEO | Triangle | 165 Hz | Brighter than LEO |
| GEO | Square | 220 Hz | Flat, stable, like the orbit |
| HEO | Sawtooth | 73 Hz | Asymmetric to match the asymmetric orbit |
| L1/L2 | Sine + 5 cent detune | 220 Hz | Beating effect — "balance point" |
Selected object boosts its regime drone +6 dB.
/moon — filter sweep + landing-site agency tones
Continuous ~80 Hz drone with low-pass filter cutoff modulated by camera longitude: near-side = warm (cutoff ~400 Hz), far-side = bright (cutoff ~3 kHz). Smooth interpolation as the camera rotates.
Landing-site selection plays the operating-agency tone (see §3.3 §/missions palette). Apollo sites = warm NASA tone. Chang'e sites = bright CNSA tone. Luna sites = low ROSCOSMOS tone.
3.3 · New 5 routes (designed v0.3 for sonification scope expansion)
/missions — agency-coloured cards
Card-tap = agency tone (see palette below). Status modulates:
- ACTIVE = continuous low ambient drone (presence)
- FLOWN = clean single tone (event)
- PLANNED = soft echoey hint (anticipation, ~300 ms decay)
Agency tone palette (used everywhere agencies appear: /missions, /moon, /mars, /iss, /tiangong, /fleet):
| Agency | Waveform | Frequency | Character |
|---|---|---|---|
| NASA | Sine + sub-octave triangle | 220 Hz + 110 Hz | Warm, anchored |
| ESA | Triangle | 277 Hz (C#4) | Bright, mid |
| JAXA | Sine + light detune | 330 Hz (E4) | Crisp, high |
| ROSCOSMOS | Square | 165 Hz | Low, declarative |
| CNSA | Triangle + sub-third | 311 Hz (Eb4) | Bright, modal |
| ISRO | Sine | 247 Hz (B3) | Mid, steady |
/science — chapter ambient texture
Continuous low pad (OscillatorNode × 2 detuned, gain ~0.05, low-pass filter cutoff ~600 Hz) that shifts character per chapter:
life-in-space= warm pad (sine + sine, no filter modulation, 110 + 165 Hz)observation= airy chime texture (triangle + sine, occasional bell-like attack envelopes, 220 + 660 Hz)mission-phases= rhythmic pulse (sine with LFO-modulated gain at 0.5 Hz, 165 Hz)basics= clean sustained tone (single sine, 220 Hz)
Chapter switch = 300 ms cross-fade between textures. Selecting a section/diagram inside a chapter = soft single tone (~80 ms envelope) at the chapter's root frequency.
/fleet — spacecraft-family chords
Spacecraft selection = 3-note chord in the operating-agency's tone palette, voiced as root + major third + perfect fifth at the agency's root frequency.
Status modulates the chord:
- ACTIVE = root note continues as low ambient drone (~0.04 gain) until selection changes
- FLOWN = clean three-note chord, 600 ms decay, then silence
- PLANNED = three-note chord with longer reverb tail (~1.2 s wet decay)
Spacecraft-class transitions (when filtering by family) play a soft swept tone matching the new family's agency.
/iss — module function tones
Module selection = tone + agency overlay:
| Module function | Waveform | Frequency | Notes |
|---|---|---|---|
| Habitation (Zvezda, Tranquility, etc.) | Sine | 110 Hz | Warm, low — "where humans live" |
| Lab (Destiny, Columbus, Kibo) | Triangle | 165 Hz | Mid bright — "where work happens" |
| Docking (Pirs, Harmony, etc.) | Sine + LFO 1 Hz | 220 Hz | Pulsing — "things attach here" |
| Power (truss segments, solar arrays) | Sawtooth low-passed | 73 Hz | Sub-bass hum — "where energy comes from" |
Visitor presence (Soyuz / Crew Dragon docked, per iss data): adds an overlaid sub-octave drone of the visiting craft's agency tone, gain 0.02.
Microgravity-axes view (microgravity-axes.ts): hovering an axis plays a soft "axis tone" (one of three: X = 220 Hz sine, Y = 277 Hz sine, Z = 330 Hz sine). Helps users distinguish which axis they're looking at without reading the label.
/tiangong — same instrument class as /iss, brighter palette
Same module-function categories as /iss but instrument family shifted:
| Module function | Waveform | Frequency | Notes |
|---|---|---|---|
| Habitation (Tianhe) | Triangle | 130 Hz | Brighter than ISS habitation |
| Lab (Wentian, Mengtian) | Sawtooth low-passed | 196 Hz | Crisper than ISS lab |
| Docking (chinarm, etc.) | Triangle + LFO 1 Hz | 261 Hz | Pulsing, brighter |
All-CNSA station ⇒ no overlaid agency tone needed (the brighter palette IS the CNSA voice in this context). Visitor presence (Shenzhou, Tianzhou) overlays Shenzhou = sine 247 Hz, Tianzhou = triangle 175 Hz at 0.02 gain.
The brighter palette + lack of multi-agency overlays makes /tiangong feel sonically distinct from /iss even though the instrument family is the same.
/mars — regime drones + agency tones + distance modulation
- Continuous ~70 Hz drone (sub-bass) modulated by Earth–Mars distance: closer = warmer (low-pass cutoff ~600 Hz), farther = darker (cutoff ~200 Hz). Camera-orientation-independent (distance is data, not viewpoint).
- Landing-site selection = operating-agency tone (per §3.3 palette). Curiosity, Perseverance, InSight = NASA. Zhurong = CNSA. Tianwen-1 lander = CNSA.
- Rover position-on-surface markers (when shown): hovering plays a discrete "scanning" texture — short bursts of band-passed noise (300 ms, attack 50 ms / release 100 ms). Distinct from the planet drone.
/ (landing) — Curator-tour intro tone
Soft single-note major-seventh chord (root 220 Hz, major third 277 Hz, fifth 330 Hz, seventh 415 Hz) with very long attack (~800 ms) and gentle release. Plays once on landing-page mount when AUDIO is enabled. Does not retrigger on every navigation back to / in the same session.
This intro tone is distinct from the PRD-016 Curator Full Tour audio (which is voice narration). The sensory tone is brief (~3 s); the Curator audio is the editorial tour. They share the chord palette but live on different toggles.
4 · Audio coexistence with PRD-016 narration
4.1 · The conflict
Both systems use the same audio output channel:
- PRD-016 narration: human-voice + TTS-voiced episodes; expects to be the focus of attention for 90 s – 8 min.
- PRD-017 sonification: continuous ambient + discrete event tones; expects to be the focus of attention nowhere — it's editorial atmosphere.
If both play at full level, the narration becomes muddy and the sonification competes for the user's interpretive attention.
4.2 · Ducking rule
When PRD-016 narration begins playing (player's play event fires):
sensoryAudio.gain.linearRampToValueAtTime(0.02, ctx.currentTime + 0.05); // ~50 ms duckWhen narration ends (ended event):
sensoryAudio.gain.linearRampToValueAtTime(1.0, ctx.currentTime + 0.20); // ~200 ms restoreWhen narration pauses (user gesture):
sensoryAudio.gain.linearRampToValueAtTime(0.5, ctx.currentTime + 0.10); // half-duck (paused, not gone)4.3 · Toggle independence
Both PRD-016 narration and PRD-017 AUDIO sub-toggle remain independently controllable. Neither force-disables the other. The ducking is an automatic gain reduction, not a state machine intervention. User can opt out of sonification by turning off the AUDIO sub-toggle; user can opt out of narration via the narration overlay.
4.4 · Player API
The two systems communicate via a tiny shared contract:
// src/lib/audio-bus.ts (new)
export type AudioBusEvent = 'narration:play' | 'narration:end' | 'narration:pause';
export const audioBus = {
emit(event: AudioBusEvent): void { /* ... */ },
on(event: AudioBusEvent, handler: () => void): () => void { /* ... */ },
};PRD-016 narration emits; PRD-017 sensory listens. No direct dependency between modules; the bus is the contract.
5 · Haptics — Capacitor + web fallback
5.1 · Path selection
import { Capacitor } from '@capacitor/core';
import { Haptics, ImpactStyle } from '@capacitor/haptics';
export async function pulse(pattern: number | number[]) {
if (Capacitor.isNativePlatform()) {
// iOS Taptic + Android-native haptics via plugin
await Haptics.impact({ style: ImpactStyle.Light });
return;
}
if ('vibrate' in navigator) {
navigator.vibrate(pattern); // Android Chrome / Firefox
return;
}
// iOS web: no haptics. Silent no-op.
}5.2 · 12 haptic patterns (from v0.1; carry forward)
| Event | Pattern (web ms) | Capacitor style |
|---|---|---|
| Recalibrate gyro | 15 | ImpactStyle.Light |
Planet selected (/explore) | 10 | ImpactStyle.Light |
Porkchop cell tap (/fly) | 12 | ImpactStyle.Light |
Solver-complete (/fly) | [10, 30, 10] | Haptics.notification({ type: NotificationType.Success }) |
Terminator crossover (/moon) | [8, 40, 8] | ImpactStyle.Medium |
Far-side reveal (/moon) | [5, 20, 5, 20, 5] | ImpactStyle.Light × 3 |
Mars-orbit crossing (/fly) | [15, 50, 15] | ImpactStyle.Medium |
Arrival event (/fly) | [10, 20, 10, 20, 10, 20, 10] | Haptics.notification({ type: NotificationType.Success }) |
Δv warning (/fly) | 30 | ImpactStyle.Heavy |
Fuel exhaustion (/fly) | [50, 30, 50] | Haptics.notification({ type: NotificationType.Warning }) |
Regime selection (/earth, /mars) | [8, 40, 8] | ImpactStyle.Medium |
| Object selection (any) | 10 | ImpactStyle.Light |
Capacitor's haptics API is coarser than web's millisecond-pattern API; the style mappings approximate the intent. Acceptable trade-off for gaining iOS support.
5.3 · iOS web limitation
iOS Safari does not implement navigator.vibrate. The HAPTIC sub-toggle is hidden on iOS web. (When wrapped under Capacitor, the path switches to @capacitor/haptics and HAPTIC works fully.)
6 · Gyroscope — camera coexistence with touch
6.1 · The path
// src/lib/gyro-camera.ts
let homeOrientation: { alpha: number; beta: number; gamma: number } | null = null;
let lastTouchEnd = 0;
window.addEventListener('deviceorientation', (e) => {
if (!gyroEnabled || prefersReducedMotion) return;
if (Date.now() - lastTouchEnd < 200) return; // T-B: pause for 200 ms after touch-end
if (!homeOrientation) {
homeOrientation = { alpha: e.alpha!, beta: e.beta!, gamma: e.gamma! };
return;
}
const dAlpha = (e.alpha! - homeOrientation.alpha) * SENSITIVITY;
// ... apply low-pass filter, dead zone, then update camera
});
canvas.addEventListener('pointerup', () => { lastTouchEnd = Date.now(); });6.2 · Tuning constants
SENSITIVITY = 0.015 rad/deg
LOW_PASS_ALPHA = 0.85 (smoothed = prev × α + raw × (1−α))
DEAD_ZONE_DEG = 2 (motion within ±2° of home is ignored — prevents jitter)
TOUCH_PAUSE_MS = 200 (gyro suppressed during + 200 ms after touch)6.3 · Per-route applicability
Gyro applies to the 7 routes with 3D scenes:
/explore— orbit camera around solar-system origin/flyheliocentric — orbit around current spacecraft position/flycislunar — orbit around Earth-Moon system origin/earth— orbit around Earth surface/moon— orbit around lunar surface/mars— orbit around Martian surface/iss— orbit around station model/tiangong— orbit around station model
/fly porkchop magnifier specifically: gyro disabled (ADR-023 conflict — magnifier uses tilt-like gestures already).
/missions, /science, /fleet — non-3D routes; gyro irrelevant; toggle has no effect on these routes (UI doesn't lie about the state).
6.4 · iOS permission
async function enableGyro(): Promise<boolean> {
const DOE = window.DeviceOrientationEvent as any;
if (typeof DOE?.requestPermission === 'function') {
const result = await DOE.requestPermission();
return result === 'granted';
}
return true; // non-iOS: permission auto-granted
}Called only on master toggle tap (P-A). Never on page load.
7 · Settings UI
7.1 · Master toggle
44×44 px button in nav, second-to-last position (between locale switcher and the menu's right edge). Icon: combined waveform-and-compass glyph (~20 px). State indication: dim outline OFF; teal-tinted fill when any sub-modality is active. Single tap toggles master ON/OFF (when ON, all enabled sub-modalities resume). Long-press (≥ 500 ms) opens settings sheet.
7.2 · Settings sheet (mobile bottom-sheet, desktop dropdown)
┌─────────────────────────────────┐
│ Sensory layer [×] │
├─────────────────────────────────┤
│ ☐ GYRO tilt to look around │ ← hidden on desktop + reduced-motion
│ ☑ AUDIO hear the physics │
│ ☐ HAPTIC feel the events │ ← hidden on iOS web (no vibrate API)
│ │
│ recalibrate gyro · triple-tap │ ← help text
└─────────────────────────────────┘Each row 44 px tall (ADR-018 touch targets). Sub-toggle state animates between on/off (≤ 150 ms).
7.3 · Visibility rules per device
| Device | GYRO row | AUDIO row | HAPTIC row |
|---|---|---|---|
| Desktop | hidden | shown | hidden |
| Mobile (Android web) | shown | shown | shown |
| Mobile (iOS web) | shown (with permission flow) | shown | hidden (no API) |
| Capacitor (Android) | shown | shown | shown |
| Capacitor (iOS) | shown | shown | shown (Capacitor haptics) |
prefers-reduced-motion (any device) | hidden | shown | hidden |
7.4 · First-time toggle-on
A non-modal toast appears 200 ms after the first toggle-on per session: "Tilt to orbit · sound on · vibration on" (text adapts to the device's available sub-modalities). Auto-dismisses after 4 s or on next tap. Never shown again that session. Not persisted across reloads (in-memory only per ADR-057).
7.5 · Recalibrate gesture
Triple-tap on the canvas re-anchors the gyro home orientation. ~150 ms teal flash on the canvas confirms the recalibration. Only works when GYRO is enabled.
8 · Performance + bundle budgets
| Constraint | Budget | Validation |
|---|---|---|
| Bundle size (audio.ts + haptics.ts + sensor.ts + sensory.ts) | < 15 KB minified gzipped | npm run build:size-check (per-module) |
| 60 fps on Pixel 5 / iPhone SE 3 across all 11 routes with sensory enabled | no drop > 5 fps median | Lighthouse mobile profile + Chrome DevTools profiler |
Sonification CPU (steady-state, busiest route = /fly heliocentric with 4 streams) | < 0.3 % | Chrome DevTools Performance pane |
| Web Audio context creation latency (cold) | < 100 ms | First-tap-to-first-tone measurement |
| Capacitor haptics call latency (Android) | < 30 ms | manual stopwatch at user-tap → vibration onset |
| Capacitor haptics call latency (iOS Taptic) | < 30 ms | same |
The Web Audio API is native (zero KB cost); only the per-route oscillator graph setup code adds bundle weight.
9 · Failure modes
| Failure | Detection | Handling |
|---|---|---|
| iOS gyro permission denied | DeviceOrientationEvent.requestPermission() returns 'denied' | GYRO sub-toggle auto-disables; AUDIO + HAPTIC remain available; settings sheet shows GYRO row but disabled with a tooltip "permission denied — enable in Safari Settings" |
| Web Audio context fails to create | exception on new AudioContext() | AUDIO sub-toggle hidden; sensory layer falls back to GYRO + HAPTIC only; logs once to console |
navigator.vibrate returns false (Android Chrome on weird device) | check return value | next pulse silently does nothing; HAPTIC sub-toggle stays available (silent failure is better than disappearing-button confusion) |
| Sonification graph has dropouts under CPU load (lower-tier device) | runtime detection via performance.now() deltas | gracefully thin to 2 streams instead of 4 on /fly; visual cue not surfaced (silent degradation) |
| User tilts past dead zone but sonification is laggy | low-pass filter α = 0.85 is the mitigation | fixed at this α; not exposed to user adjustment in v1 |
10 · Testing
10.1 · Manual test matrix (every device per release)
| Device | Browser / wrapper | Tests |
|---|---|---|
| iPhone SE 3 | Safari | iOS gyro permission flow; AUDIO on built-in speaker; no haptics expected |
| iPhone 14 Pro | Capacitor wrapper | iOS gyro w/o permission prompt (Info.plist); Capacitor haptics; AUDIO |
| Pixel 5 | Chrome | Android gyro; navigator.vibrate haptics; AUDIO; reduced-motion fallback |
| Pixel 5 | Capacitor wrapper | Same as Chrome but via Capacitor haptics path |
| Galaxy A-series | Chrome | mid-range Android perf check (60 fps target) |
| MacBook (any model) | Chrome | desktop AUDIO only; GYRO + HAPTIC hidden |
| MacBook | Firefox | desktop AUDIO only on Firefox WebAudio impl |
10.2 · Coexistence tests
- Sonification ducking with PRD-016 narration: turn narration on → measure sonification gain drop within 50 ms of narration play event; turn narration off → measure restore within 200 ms.
- Sonification × screen reader: enable VoiceOver / TalkBack → sonification gain → 0; disable → restored.
- Gyro × touch coexistence: drag canvas → gyro suppressed; release touch → gyro resumes from new position after 200 ms.
10.3 · Listening-quality reviews (per design)
For each of the 11 routes, a listening session with Marko + 1 reviewer to confirm:
- Sonic identity is recognisable per route
- No clipping or distortion on built-in speakers
- Mix is musical, not noisy
- Headphone-vs-speaker delta is acceptable
11 · Resolved decisions + open questions
Resolved 2026-05-16:
- Sonification scope — RESOLVED: All 11 routes (not just original 6). 5 new per-route designs added in §3.3.
- Capacitor haptics — RESOLVED:
@capacitor/hapticsin Capacitor builds,navigator.vibratefallback on web. iOS gets basic Taptic when wrapped; iOS web has no haptics. - Audio coexistence with narration — RESOLVED: Sonification ducks to ~0.02 gain while narration plays. Toggles independent. §4.
- Settings UI — RESOLVED: Two separate buttons in nav (narration + sensory). Each its own bottom-sheet.
- Storage — RESOLVED: in-memory only. ADR-057 forbids localStorage; the original RFC's "v0.4 reconsider localStorage" line is deleted.
- Six anchor design choices (G-C, T-B, S-C, I-B, U-2, P-A) — RESOLVED: all stand. Mature design from v0.1; no rework.
Additional decisions resolved 2026-05-16 (second batch):
- Default master-toggle state — RESOLVED: Default OFF + one-time landing-page hint. "Tilt + sound + buzz — try sensory mode" dismissable toast on
/. prefers-reduced-motion× AUDIO — RESOLVED: AUDIO remains user-controllable under reduced-motion. Only GYRO + HAPTIC are hidden.- Recalibrate gesture — RESOLVED: Triple-tap on canvas (150 ms teal flash confirms). iOS-accessibility-shortcut collision is an implementation-time fallback to a dedicated settings-sheet button.
Remaining operational follow-ups:
- Sonification listening reviews for the 5 new route designs (§3.3). Paper designs; need a session each with Marko + 1 reviewer before ship.
- iPhone SE 3 audio quality on built-in speaker. Kepler chord (8 oscillators on small driver). Tune oscillator gains down if muddy at implementation time.
- Per-screen sonification SCALE_EXP tuning. §3.1 sets
0.5for the Kepler chord. Other routes don't currently use SCALE_EXP. Confirm at implementation; expose more constants only if listening reviews surface a need.
RFC-020 · Orrery · Sensory Layer · Drafted 2026-05-16 · Closes-into-PRD-017