Skip to content

ADR-058: Additive pyannote Diarization

  • Status: Accepted (amended 2026-05-30 — dependency packaging and defaults; see Amendment)
  • Date: 2026-04-03
  • Authors: Podcast Scraper Team
  • Related RFCs: RFC-058
  • Related PRDs: PRD-020

Context & Problem Statement

The current screenplay formatting uses a time-gap heuristic to assign speakers: if a silence gap exceeds gap_s, the next segment is attributed to a different speaker via round-robin cycling. This produces systematically wrong attribution in rapid exchanges, same-speaker pauses, and multi-guest panels.

Neural speaker diarization (voice-embedding-driven "who said what") solves this, but there are multiple integration strategies: replace Whisper entirely with WhisperX, add pyannote as a second pass after Whisper, or use cloud diarization APIs.

Decision

We add pyannote.audio as an additive second pass after Whisper transcription, with diarization dependencies in [ml] / [dev] and lazy imports (see amendment).

  1. Additive, not replacement: Whisper transcription is preserved as-is. pyannote runs after Whisper to produce speaker timelines, which are aligned to Whisper segments via maximum-overlap matching. The existing gap-based path remains the fallback when diarization is disabled or fails at runtime.
  2. Segment-level diarization: One speaker is assigned per Whisper segment (not per word). This matches the current screenplay format and avoids the complexity of forced word-level alignment.
  3. Waveform loading via torchaudio: Audio is loaded with torchaudio.load() and passed as a waveform to pyannote, avoiding a known 4x performance penalty when passing file paths (pyannote issue #1702).
  4. Dependencies in [ml] and [dev] (amended): pyannote.audio and torchaudio ship in the [ml] extra (pipeline/Docker) and are pinned in [dev] for CI/dev venv parity. Lazy import at function level (ADR-005) so the package loads normally when pyannote is not installed or diarization is off.

Rationale

  • Additive: Lower integration risk than replacing the entire Whisper pipeline with WhisperX. All existing Whisper code paths, tests, and behaviors are preserved. If pyannote has issues, the gap-based fallback works.
  • Segment-level: Simpler alignment algorithm, matches current screenplay format. Word-level diarization requires forced alignment (WhisperX territory) and can be added later if needed.
  • Waveform loading: Measured 12s vs 50s for 3-minute clips. This is a straightforward performance optimization that should always be used.
  • Separate extra: Originally rejected merging into [ml] to avoid penalizing every ML install (see Amendment).

Alternatives Considered

  1. WhisperX as full pipeline replacement: Rejected; replaces proven Whisper integration, slightly lower diarization accuracy (~5%), larger blast radius. Can be evaluated after diarization proves value.
  2. Voice-activity-based speaker change detection (Silero VAD): Rejected; still a heuristic with no voice identity. Only marginally better than gap-based rotation.
  3. Cloud diarization APIs (Google, AssemblyAI): Rejected; per-minute costs, vendor lock-in, no offline support. Conflicts with local-first philosophy.
  4. Merge [diarize] into [ml] / [dev]: Originally rejected at acceptance time; revised at implementation — see Amendment.

Consequences

  • Positive: Accurate, voice-based speaker attribution. Auto speaker count detection (eliminates manual screenplay_num_speakers). Multi-speaker panels correctly handled. Downstream quality improves for GIL quotes and KG speaker nodes.
  • Negative: HuggingFace token required (gated model). GPU strongly recommended (CPU ~8.5 min vs GPU ~1.5 min for 60 min audio). Heavier [ml] / [dev] installs.
  • Neutral: diarize=true is the default for local Whisper paths; API transcription providers coerce diarize=false. Gap-based screenplay remains the runtime fallback when diarization fails.

Implementation Notes

  • Module: src/podcast_scraper/providers/ml/diarization/
  • Protocol: DiarizationProvider (PEP 544) with diarize() method
  • Alignment: Maximum-overlap matching between Whisper segments and diarization timeline, with carry-forward for gaps
  • Caching: Diarization results cached by sha256(audio_content) + diarization_config_hash in .cache/diarization/
  • Config: diarize: bool = True (coerced off for non-local-whisper providers), hf_token, num_speakers, min_speakers, max_speakers, diarization_device, diarization_model. Eligible providers: whisper, tailnet_dgx_whisper.
  • Relationship to ADR-005: Follows lazy ML dependency loading pattern
  • Relationship to ADR-036: Preprocessed audio (RFC-040) feeds into diarization

References

Amendment (2026-05-30)

At implementation (Wave 2 / #482), two acceptance-time choices were revised:

  1. No separate [diarize] extra. pyannote.audio>=3.1 and torchaudio are bundled in [ml] and pinned in [dev] (pyproject.toml). Install hint: pip install -e '.[ml]'. Documented in docs/guides/DEPENDENCIES_GUIDE.md.
  2. diarize defaults to true for local Whisper transcription (whisper and tailnet_dgx_whisper). Cloud/API providers (OpenAI, Gemini, Deepgram, etc.) coerce diarize=false at config validation. --no-diarize remains available.

Original filename (…-with-separate-extra.md) is retained for link stability.