Skip to content

RFC-058: Audio-Based Speaker Diarization

  • Status: Completed (v2.6.0–v2.7) — pyannote provider + alignment shipped (src/podcast_scraper/diarization/), ADR-058 ratified, providers wired into pipeline + evaluation harness. WhisperX comparison eval is future scope, not a gating deliverable.
  • Update (2026-07-11, #1170): the default pipeline was promoted from speaker-diarization-3.1 to speaker-diarization-community-1 (v4) — it beats 3.1 on the full v3 fixture set (count 40/45, DER 7.1%) and is non-gated. 3.1 is now the registry fallback. Historical 3.1 snippets below are preserved as the original design. See EVAL_DIARIZATION_31_VS_COMMUNITY1_RTTM_2026_07.
  • Authors: Architecture Review
  • Stakeholders: Core Pipeline, ML Providers, Transcription Consumers
  • Related PRDs:
  • docs/prd/PRD-020-audio-speaker-diarization.md
  • docs/prd/PRD-002-whisper-fallback.md
  • docs/prd/PRD-008-speaker-name-detection.md
  • Related RFCs:
  • docs/rfc/RFC-005-whisper-integration.md (Whisper lifecycle)
  • docs/rfc/RFC-006-screenplay-formatting.md (gap-based rotation — to be superseded)
  • docs/rfc/RFC-010-speaker-name-detection.md (NER name pipeline — preserved)
  • docs/rfc/RFC-040-audio-preprocessing-pipeline.md (FFmpeg preprocessing — feeds into diarization)
  • docs/rfc/RFC-059-speaker-detection-refactor-test-audio.md (companion: refactor + test improvements)
  • docs/rfc/RFC-060-diarization-aware-commercial-cleaning.md (downstream: diarization-aware sponsor detection)
  • Related Issues:
  • Issue #414: Audio Pipeline Separation
  • Related Documents:
  • docs/architecture/ARCHITECTURE.md (pipeline flow)

Abstract

This RFC defines the technical design for adding neural speaker diarization to the local Whisper transcription pipeline via pyannote.audio. Diarization replaces the gap-based round-robin speaker rotation in format_screenplay_from_segments() with voice-embedding-driven speaker attribution, producing accurate "who said what" transcripts. diarize defaults to on for local Whisper paths (whisper, tailnet_dgx_whisper); API transcription providers coerce it off. Gap-based screenplay remains the runtime fallback when diarization is unavailable. The NER name-mapping system from RFC-010 assigns real names to diarized speaker IDs.

Architecture Alignment: Diarization is introduced as an optional post-transcription stage in episode_processor.py, sitting between Whisper transcription and screenplay formatting. This follows the pipeline-stage pattern established by RFC-040 (audio preprocessing) and aligns with Issue #414's vision of separating audio processing into distinct stages.

Problem Statement

The current screenplay formatting (format_screenplay_from_segments() in ml_provider.py) uses a time-gap heuristic to assign speakers:

if prev_end is not None and start - prev_end > gap_s:
    current_speaker_idx = (current_speaker_idx + 1) % max(
        config.MIN_NUM_SPEAKERS, num_speakers
    )

This produces systematically wrong speaker attribution:

  1. Rapid exchanges — When speakers alternate faster than gap_s, both turns are attributed to the same speaker
  2. Same-speaker pauses — A long pause (thinking, drinking water) triggers a false speaker switch
  3. Multi-guest panels — Round-robin cycling through 3+ speakers produces nonsensical attribution order
  4. Overlapping speech — Cross-talk is always attributed to the current speaker index

These errors propagate into downstream features (GIL quote extraction, KG speaker nodes) and make screenplay transcripts unreliable for research or analysis.

Use Cases:

  1. Interview transcript accuracy: Host and guest lines correctly attributed during rapid back-and-forth
  2. Panel discussion: 3-4 speakers correctly identified and tracked throughout the episode
  3. Quote extraction: GIL/KG downstream features receive reliably attributed speaker segments

Goals

  1. Accurate diarization: Replace gap-based rotation with neural speaker embeddings (target: < 15% DER)
  2. Transparent integration: Diarization as an optional pipeline stage, not a replacement for Whisper
  3. Name preservation: Map diarized speaker IDs to NER-detected names (RFC-010 pipeline intact)
  4. Backward compatibility: diarize=true default for local Whisper; API providers coerce off; gap-based fallback on diarization failure
  5. Cacheable: Diarization results cached by audio content hash to avoid re-processing

Constraints & Assumptions

Constraints:

  • pyannote.audio models are gated on HuggingFace — requires user to accept model terms and provide access token
  • GPU strongly recommended (CPU: ~8.5 min for 60 min audio; GPU with proper waveform loading: ~1.5 min)
  • Only applies to local Whisper path — cloud providers (OpenAI, Gemini, Mistral APIs) transcribe without local audio access
  • Must not break API transcription paths when diarization is coerced off; gap fallback must remain available
  • pyannote adds significant dependency weight: speechbrain, torchaudio, HF model downloads

Assumptions:

  • Users opting into --diarize accept the HuggingFace token requirement
  • Podcast audio quality is generally sufficient for speaker embedding extraction
  • Most podcast episodes have 2-4 speakers; maximum 20 is a safe upper bound
  • Segment-level diarization (not word-level) is sufficient for the initial implementation

Design & Implementation

1. New Module: diarization/

Create a new provider-style module under the ML providers directory:

src/podcast_scraper/
├── providers/
│   └── ml/
│       ├── ml_provider.py          # existing — gains diarize_and_format() method
│       ├── speaker_detection.py    # existing — NER pipeline unchanged
│       └── diarization/            # NEW
│           ├── __init__.py
│           ├── base.py             # DiarizationProvider protocol
│           ├── pyannote_provider.py # pyannote.audio implementation
│           └── alignment.py        # Whisper segments ↔ diarization timeline alignment

2. DiarizationProvider Protocol

from typing import Protocol, List, Tuple, Optional

class DiarizationSegment:
    """A diarized segment with speaker ID and time range."""
    start: float           # seconds
    end: float             # seconds
    speaker: str           # e.g. "SPEAKER_00"

class DiarizationResult:
    """Complete diarization output."""
    segments: List[DiarizationSegment]
    num_speakers: int
    model_name: str

class DiarizationProvider(Protocol):
    """Protocol for speaker diarization backends."""

    def diarize(
        self,
        audio_path: str,
        *,
        num_speakers: Optional[int] = None,
        min_speakers: int = 2,
        max_speakers: int = 20,
    ) -> DiarizationResult:
        """Run speaker diarization on an audio file.

        Args:
            audio_path: Path to audio file (WAV, MP3, etc.)
            num_speakers: Known number of speakers (None = auto-detect)
            min_speakers: Minimum speakers for auto-detection
            max_speakers: Maximum speakers for auto-detection

        Returns:
            DiarizationResult with speaker-attributed segments
        """
        ...

3. pyannote.audio Implementation

import torch
import torchaudio
from pyannote.audio import Pipeline

class PyAnnoteDiarizationProvider:
    """Speaker diarization using pyannote.audio community pipeline."""

    def __init__(
        self,
        hf_token: str,
        device: str = "auto",
        model_name: str = "pyannote/speaker-diarization-3.1",
    ):
        self._pipeline = Pipeline.from_pretrained(model_name, use_auth_token=hf_token)

        if device == "auto":
            device = "cuda" if torch.cuda.is_available() else "cpu"
        self._pipeline.to(torch.device(device))

    def diarize(self, audio_path, *, num_speakers=None, min_speakers=2, max_speakers=20):
        # Load audio as waveform (avoids pyannote's slow file-path loading)
        waveform, sample_rate = torchaudio.load(audio_path)

        params = {}
        if num_speakers is not None:
            params["num_speakers"] = num_speakers
        else:
            params["min_speakers"] = min_speakers
            params["max_speakers"] = max_speakers

        diarization = self._pipeline(
            {"waveform": waveform, "sample_rate": sample_rate},
            **params,
        )

        segments = []
        for turn, _, speaker in diarization.itertracks(yield_label=True):
            segments.append(DiarizationSegment(
                start=turn.start,
                end=turn.end,
                speaker=speaker,
            ))

        unique_speakers = {s.speaker for s in segments}
        return DiarizationResult(
            segments=segments,
            num_speakers=len(unique_speakers),
            model_name=self._pipeline.model_name,
        )

Key implementation detail: loading audio via torchaudio.load() and passing the waveform directly to pyannote avoids a known 4x performance penalty when passing file paths (see pyannote issue #1702).

4. Alignment Algorithm

The alignment module maps Whisper transcription segments to diarization speaker IDs:

def align_segments_to_speakers(
    whisper_segments: List[dict],
    diarization: DiarizationResult,
) -> List[Tuple[dict, str]]:
    """Assign a speaker ID to each Whisper segment based on diarization overlap.

    For each Whisper segment, find the diarization segment(s) that overlap it,
    then assign the speaker with the greatest overlap duration.

    Args:
        whisper_segments: Whisper segments with start/end/text
        diarization: DiarizationResult from diarization provider

    Returns:
        List of (whisper_segment, speaker_id) tuples
    """

Algorithm:

  1. Build an interval index from diarization segments (for O(log n) lookups)
  2. For each Whisper segment [ws, we]: a. Find all diarization segments that overlap [ws, we] b. Compute overlap duration with each: overlap = min(we, de) - max(ws, ds) c. Assign the speaker with the maximum total overlap d. If no overlap found (gap in diarization), carry forward the previous speaker
  3. Return aligned (segment, speaker_id) pairs

Edge cases:

  • No diarization overlap: Carry forward previous speaker (silence gaps between diarization segments)
  • Multiple speakers per Whisper segment: Assign majority speaker (segment-level, not word-level)
  • Whisper segment extends beyond diarization: Clip to diarization boundary

5. Integration into Pipeline

In episode_processor.py:

def _diarize_and_format_screenplay(
    self,
    audio_path: str,
    whisper_segments: List[dict],
    speaker_names: List[str],
    cfg: Config,
) -> str:
    """Diarize audio and format screenplay with real speaker attribution."""

    # 1. Run diarization
    diarization_result = self._diarization_provider.diarize(
        audio_path,
        num_speakers=cfg.num_speakers,
        min_speakers=cfg.min_speakers,
        max_speakers=cfg.max_speakers,
    )

    # 2. Align Whisper segments to diarized speakers
    aligned = align_segments_to_speakers(whisper_segments, diarization_result)

    # 3. Map speaker IDs to detected names
    speaker_map = self._map_speakers_to_names(
        diarization_result, speaker_names
    )

    # 4. Format screenplay
    return self._format_diarized_screenplay(aligned, speaker_map)

Pipeline flow with diarization enabled:

Download → [Preprocess (RFC-040)] → Whisper transcribe
                                        ↓
                                  segments + text
                                        ↓
                            ┌──── diarize=true? ────┐
                            │                       │
                       YES: pyannote               NO: gap-based
                       diarize audio               rotation (RFC-006)
                            │                       │
                       align segments               │
                       to speakers                  │
                            │                       │
                            └───────────────────────┘
                                        ↓
                              NER name mapping (RFC-010)
                                        ↓
                              Screenplay output

6. Speaker Name Mapping

Diarization produces anonymous IDs (SPEAKER_00, SPEAKER_01, ...). The NER pipeline (RFC-010) produces names. Mapping strategy:

  1. Intro heuristic: The speaker with the most speaking time in the first 90 seconds → likely host → map to first detected host name
  2. Speaking time rank: Remaining speakers ordered by total speaking time → map to remaining names in order
  3. Fallback: If NER detected fewer names than diarized speakers, use SPEAKER_N for unmapped speakers
def _map_speakers_to_names(
    self,
    diarization: DiarizationResult,
    detected_names: List[str],
) -> Dict[str, str]:
    """Map diarized speaker IDs to detected names.

    Strategy: speaker with most time in first 90s → host name;
    remaining by total speaking time → remaining names in order.
    """

7. Configuration

New config fields in Config (Pydantic model):

# Diarization settings
diarize: bool = True  # coerced False for API transcription providers
hf_token: Optional[str] = None  # also: HF_TOKEN env var, ~/.huggingface/token
diarization_num_speakers: Optional[int] = None  # None = auto-detect
diarization_min_speakers: int = 2
diarization_max_speakers: int = 20
diarization_device: str = "auto"  # "cpu", "cuda", "mps"
diarization_model: str = "pyannote/speaker-diarization-community-1"  # 3.1 = fallback (#1170)

CLI flags:

--diarize / --no-diarize     Enable/disable speaker diarization
--hf-token TEXT              HuggingFace access token
--num-speakers INT           Override auto-detected speaker count
--min-speakers INT           Minimum speakers (default: 2)
--max-speakers INT           Maximum speakers (default: 20)
--diarization-device TEXT    Device for diarization (auto/cpu/cuda/mps)

8. Caching

Diarization results are expensive to compute. Cache alongside transcript cache:

  • Cache key: sha256(audio_content) + diarization_config_hash
  • Cache value: Serialized DiarizationResult (JSON)
  • Location: diarization_cache_dir (defaults to <output_dir>/.cache/diarization/)
  • Invalidation: Audio content change or diarization config change

9. Dependency Management

Add pyannote.audio as a new optional extra in pyproject.toml:

[project.optional-dependencies]
ml = [
    # ... existing ml deps ...
    "pyannote.audio>=3.1",
    "torchaudio",
]
dev = [
    # ... existing dev deps; pyannote pins aligned with [ml] for CI parity ...
]

Install with:

pip install -e ".[ml]"
# dev/CI venv:
pip install -e ".[dev,ml]"

Lazy import pattern (same as Whisper):

def _import_pyannote():
    try:
        from pyannote.audio import Pipeline
        return Pipeline
    except ImportError:
        raise ProviderDependencyError(
            "pyannote.audio is required for speaker diarization. "
            "Install with: pip install -e '.[ml]'"
        )

Key Decisions

  1. Additive pyannote (not WhisperX replacement)
  2. Decision: Keep existing Whisper integration, add pyannote as a second pass

  3. Rationale: Lower integration risk; preserves all existing Whisper code paths and tests; best diarization accuracy. WhisperX can be evaluated as a future optimization (see PRD-020 DC1)

  4. Segment-level (not word-level) diarization

  5. Decision: Assign one speaker per Whisper segment, not per word

  6. Rationale: Simpler alignment; matches current screenplay format; word-level requires forced alignment (WhisperX territory). Can be added later

  7. Waveform loading via torchaudio

  8. Decision: Load audio with torchaudio.load() before passing to pyannote

  9. Rationale: pyannote's file-path loading has a known 4x performance penalty (issue #1702). Waveform loading achieves 12s vs 50s for 3-min clips

  10. Dependencies in [ml] and [dev] (not a separate extra)

  11. Decision: Bundle pyannote.audio and torchaudio in [ml]; pin in [dev] for CI parity

  12. Rationale: Operator choice at implementation — avoids a fourth optional extra; lazy imports keep non-diarization paths light at runtime. See ADR-058 amendment (2026-05-30).

  13. HuggingFace token from multiple sources

  14. Decision: Accept via --hf-token, HF_TOKEN env var, or ~/.huggingface/token
  15. Rationale: Follows HuggingFace ecosystem conventions; flexible for CI and local use

Alternatives Considered

  1. WhisperX as full pipeline replacement
  2. Description: Replace current Whisper integration entirely with WhisperX
  3. Pros: Single pipeline, batched inference (70x realtime), word-level timestamps, VAD pre-filtering
  4. Cons: Replaces proven Whisper integration; slightly lower diarization accuracy (~5%); larger blast radius

  5. Why Deferred: Too risky for initial implementation. Evaluate after diarization proves value

  6. Simple voice-activity-based speaker change detection

  7. Description: Use Silero VAD to detect speech segments and infer speaker changes from silence patterns
  8. Pros: Lightweight, no HuggingFace token needed
  9. Cons: Not actually diarization — still a heuristic; no voice embeddings; only marginally better than current gap-based approach

  10. Why Rejected: Doesn't solve the core problem (no voice identity)

  11. Cloud diarization APIs (e.g., Google Speech-to-Text, AssemblyAI)

  12. Description: Use cloud APIs that include diarization
  13. Pros: No local GPU needed; high accuracy
  14. Cons: Per-minute costs; requires network; vendor lock-in; doesn't work offline
  15. Why Rejected: Conflicts with project philosophy of local-first, optional cloud

Testing Strategy

Test Coverage:

  • Unit tests: Alignment algorithm, speaker name mapping, cache key generation, config validation
  • Integration tests: mocked diarization path (patches _create_pyannote_pipeline + _load_waveform; no real pyannote / HF token) — see tests/integration/providers/ml/test_diarization.py. Real pyannote runs only in the e2e tier.
  • E2E tests: Full pipeline with --diarize on sample podcast audio

Test Organization:

  • tests/unit/podcast_scraper/providers/ml/diarization/ — alignment, mapping, caching
  • tests/integration/providers/ml/test_diarization.py — mocked diarization (no real pyannote, no HF token; markers integration, diarization)
  • tests/e2e/test_diarization_e2e.py — real pyannote on v2 fixture audio (e2e tier)
  • Marker: @pytest.mark.diarization (integration runs are mocked; real pyannote + HuggingFace token live in the e2e test)
  • Marker: @pytest.mark.slow (diarization is inherently slow)

Test Fixtures:

  • RFC-059 shipped unique voices per speaker (v2 SPEAKER_VOICE_MAP), so pyannote separates speakers in the TTS-generated fixtures — tests/e2e/test_diarization_e2e.py passes 4/4 on v2
  • Real podcast audio snippets are not required: the v2 fixtures exercise the real-pyannote e2e path

Test Execution:

  • Unit tests: make test (fast, no ML dependencies)
  • Integration tests: make test-slow or dedicated make test-diarize target
  • CI: Skip diarization tests by default (no GPU, no HuggingFace token); run in dedicated slow CI job

Rollout & Monitoring

Rollout Plan:

  • Phase 1: Core diarization provider + alignment algorithm + unit tests
  • Phase 2: Pipeline integration + caching + CLI flags + config
  • Phase 3: Integration tests with real/improved test audio + benchmark on 10+ episodes
  • Phase 4: Documentation, HuggingFace setup guide, examples

Monitoring:

  • Diarization Error Rate (DER) measured on benchmark episodes
  • Processing time logged per episode (diarization step isolated)
  • Cache hit rate tracked in pipeline metrics

Success Criteria:

  1. DER < 15% on benchmark episodes
  2. Screenplay attribution >= 85% correct on manual spot-check
  3. Processing overhead < 2x vs Whisper-only (with GPU)
  4. API transcription paths coerce diarize=false; gap fallback when diarization unavailable

Relationship to Other RFCs

This RFC (RFC-058) is part of the audio pipeline evolution that includes:

  1. RFC-006: Screenplay Formatting — Current gap-based system; RFC-058 provides an alternative path
  2. RFC-010: Speaker Name Detection — NER pipeline preserved; names map to diarized speaker IDs
  3. RFC-040: Audio Preprocessing — Preprocessed audio feeds into diarization
  4. RFC-059: Speaker Detection Refactor & Test Audio — Companion RFC; refactors speaker detection module and improves test audio to make diarization testable

Key Distinction:

  • RFC-058 (this): Adds the diarization capability (pyannote, alignment, pipeline integration)
  • RFC-059: Improves the infrastructure that diarization depends on (modular speaker detection, test audio with distinct voices)

Together, these RFCs deliver:

  • Accurate, voice-based speaker attribution in transcripts
  • Clean, modular speaker detection architecture ready for diarization integration
  • Test fixtures capable of validating diarization quality

Benefits

  1. Accurate speaker attribution: Neural voice embeddings replace blind gap-based rotation
  2. Auto speaker count: Eliminates manual screenplay_num_speakers configuration
  3. Multi-speaker support: Correctly handles panels with 3+ speakers
  4. Downstream quality: Improves GIL quote attribution and KG speaker nodes
  5. Opt-in safety: Zero impact on existing users and workflows

Migration Path

  1. Phase 1 (shipped): diarize=true default for local Whisper; API providers coerce off
  2. Phase 2: --no-diarize for operators who want gap-based screenplay only
  3. Phase 3: Diarization result caching (RFC Phase 4) and fixture-backed integration proof (#111)
  4. Future: Evaluate WhisperX for bundled pipeline if speed/word-level timestamps become priorities

Open Questions

  1. ~~Should [diarize] be merged into [ml] extra or remain separate?~~ Resolved: bundled in [ml] / [dev] (ADR-058 amendment 2026-05-30).
  2. Should the alignment use an interval tree library (e.g., intervaltree) or a simple sorted-list scan?
  3. How to handle the HuggingFace model acceptance UX — CLI wizard on first run?
  4. Should diarization confidence scores be exposed in output metadata?
  5. What real podcast audio (CC-licensed) to use for integration test benchmarks?

References

  • Related PRD: docs/prd/PRD-020-audio-speaker-diarization.md
  • Related Issues: #414 (Audio Pipeline Separation)
  • Source Code: src/podcast_scraper/providers/ml/ml_provider.py (screenplay formatting)
  • Source Code: src/podcast_scraper/providers/ml/speaker_detection.py (NER pipeline)
  • External: pyannote/pyannote-audio (v4.0.4)
  • External: m-bain/whisperX (v3.3.2, future option)
  • External: pyannote issue #1702 (waveform loading performance)