Skip to content

Critical Path Testing Guide

See also:

This guide focuses on what to test and how to prioritize. Use the @pytest.mark.critical_path marker for tests that validate the critical path - these run in the fast test suite.

What is the Critical Path?

Critical Path = Full Workflow with All Core Features

The critical path represents the essence of this project - the complete workflow that delivers the main value:

RSS → Parse → Download/Transcribe → Speaker Detection → Summarization → Metadata → Files

Path 1: Transcript Download (when transcript URL exists)

ML Provider Flow:

  • RSS → Parse → Download Transcript → NER Speaker Detection → Local Summarization → Metadata → Files

OpenAI Provider Flow:

  • RSS → Parse → Download Transcript → OpenAI Speaker Detection → OpenAI Summarization → Metadata → Files

Path 2: Transcription (when transcript URL missing)

ML Provider Flow:

  • RSS → Parse → Download Audio/Video → Whisper Transcription → NER Speaker Detection → Local Summarization → Metadata → Files

OpenAI Provider Flow:

  • RSS → Parse → Download Audio/Video → OpenAI Transcription → OpenAI Speaker Detection → OpenAI Summarization → Metadata → Files

Why All Paths Matter:

  • All variants are valid configurations users can choose
  • All are essential for the tool to work correctly with different provider choices
  • All must be tested to ensure complete coverage regardless of provider selection

Why Speaker Detection and Summarization are Critical

  • Speaker Detection: Core feature that identifies hosts and guests from metadata
  • ML Provider: Uses spaCy NER models (local)
  • OpenAI Provider: Uses OpenAI API (cloud)
  • Summarization: Core feature that generates episode summaries from transcripts
  • ML Provider: Uses Transformers models (local)
  • OpenAI Provider: Uses OpenAI API (cloud)
  • Both are essential: They represent the main value proposition of the project
  • Both provider types are critical: Users can choose ML (local, free) or OpenAI (cloud, paid) based on their needs

Critical Path and Test Pyramid

The critical path should be tested at all three levels of the test pyramid:

        /\
       /E2E\          ← Complete critical path workflows (user-facing)
      /------\
     /Integration\    ← Critical path component interactions
    /------------\
   /    Unit      \   ← Critical path individual functions
  /----------------\

Unit Tests (Bottom Layer)

What to Test:

  • Individual functions in the critical path
  • Each step in isolation: RSS parsing, transcript download, transcription, NER, summarization, metadata generation

Examples:

  • rss_parser.py: Parse RSS feeds correctly
  • rss/downloader.py: Download transcripts/audio correctly
  • providers/ml/whisper_utils.py: Transcribe audio correctly
  • speaker_detection.py: Detect speakers from text
  • providers/ml/summarizer.py: Generate summaries from transcripts
  • metadata.py: Generate metadata files

Priority: HIGH - These are the building blocks of the critical path

Integration Tests (Middle Layer)

What to Test:

  • Component interactions along the critical path
  • How components work together: RSS → Episode → Provider → File

Critical Path Integration Tests:

  1. test_full_workflow_with_ner_and_summarization ⭐ ESSENTIAL (ML Providers)
  2. Validates: RSS → Parse → Download/Transcribe → NER → Local Summarization → Metadata → Files
  3. Uses: ML providers (Whisper, spaCy NER, Transformers)

  4. This is the complete critical path with ML providers

  5. test_full_workflow_with_openai_providers ⭐ ESSENTIAL (OpenAI Providers)

  6. Validates: RSS → Parse → Download/Transcribe → OpenAI Speaker Detection → OpenAI Summarization → Metadata → Files

  7. Uses: OpenAI providers (mocked API calls)

  8. This is the complete critical path with OpenAI providers

  9. test_critical_path_with_real_models (ML Providers with Real Models)

  10. Validates: Critical path with real cached ML models

  11. Uses: Real Whisper, spaCy, Transformers models (cached)

  12. test_critical_path_with_openai_providers (OpenAI Providers)

  13. Validates: Critical path with OpenAI providers (mocked API)

  14. Uses: OpenAI providers for all three services

  15. test_rss_to_metadata_generation

  16. Validates: RSS → Parse → Download Transcript → Metadata → Files

  17. Covers Path 1 (transcript download)

  18. test_rss_to_transcription_workflow

  19. Validates: RSS → Parse → Download Audio → Transcription → Metadata → Files

  20. Covers Path 2 (transcription)

  21. test_episode_processor_audio_download_and_transcription

  22. Validates: Episode processor functions for audio download and transcription

  23. test_speaker_detection_in_transcription_workflow

  24. Validates: RSS → Parse → Download Audio → Transcribe → Speaker Detection → Metadata → Files

Priority: CRITICAL - These validate the critical path works end-to-end

E2E Tests (Top Layer)

What to Test:

  • Complete user workflows for the critical path
  • All three entry points: CLI, Library API, Service API

Critical Path E2E Tests:

  1. test_cli_basic_transcript_download (CLI)
  2. Validates: CLI transcription end-to-end

  3. Uses: --transcribe-missing

  4. test_library_api_basic_pipeline (Library API)

  5. Validates: run_pipeline(config) transcription end-to-end

  6. Uses: transcribe_missing=True

  7. test_service_api_basic_run (Service API)

  8. Validates: service.run(config) transcription end-to-end
  9. Uses: transcribe_missing=True

Priority: CRITICAL - These validate users can actually use the tool

Decision Framework: Is This Critical Path?

When deciding what to test, ask:

  1. Is this part of the critical path?

  2. ✅ RSS parsing → Test it

  3. ✅ Transcript download/transcription → Test it
  4. ✅ NER speaker detection → Test it
  5. ✅ Summarization → Test it
  6. ✅ Metadata generation → Test it
  7. ❌ Extended features → Lower priority

  8. ❌ Configuration edge cases → Lower priority

  9. What test level should I use?

  10. Unit: Testing individual functions → Unit test

  11. Integration: Testing component interactions → Integration test

  12. E2E: Testing complete user workflow → E2E test

  13. Should this be fast or slow?

  14. Fast: Critical path tests that use mocked components → Fast

  15. Fast: Critical path tests with OpenAI providers (mocked API) → Fast
  16. Slow: Critical path tests that use real ML models → Slow (but still critical)
  17. Slow: Non-critical path tests → Slow

Test Prioritization

Priority 1: Critical Path (Must Have)

These tests MUST exist and MUST pass:

  • ✅ Full workflow integration test with NER and summarization
  • ✅ Integration tests for both paths (transcript download + transcription)
  • ✅ E2E tests for all three entry points (CLI, Library API, Service API)
  • ✅ Unit tests for all critical path functions

Execution: Run in fast test suite (with mocks) for quick feedback

Priority 2: Critical Path with Real Models (Should Have)

These tests validate critical path with real implementations:

  • ✅ E2E tests with real Whisper transcription
  • ✅ E2E tests with real NER models
  • ✅ E2E tests with real summarization models

Execution: Run in slow test suite (with real models) for comprehensive validation

Priority 3: Extended Features (Nice to Have)

These tests validate features beyond the critical path:

  • ⚠️ Extended metadata features
  • ⚠️ Configuration edge cases
  • ⚠️ Error handling edge cases
  • ⚠️ HTTP behavior tests

Execution: Run in slow test suite

Critical Path Test Coverage Matrix

Scenario Unit Tests Integration Tests E2E Tests
RSS parsing ✅ Parse RSS feeds ✅ RSS → Episode ✅ Full workflow
Transcript download ✅ Download logic ✅ Download → Metadata ✅ Full workflow
Transcription ✅ Whisper integration ✅ Audio → Transcript ✅ Full workflow
NER (Speaker Detection) ✅ NER extraction ✅ Transcript → Speakers ✅ Full workflow
Summarization ✅ Summary generation ✅ Transcript → Summary ✅ Full workflow
Metadata generation ✅ Metadata creation ✅ All data → Metadata ✅ Full workflow

Current Critical Path Coverage

Integration-Fast Tests ✅

6 tests covering critical path:

  1. test_full_workflow_with_ner_and_summarization ⭐ ESSENTIAL (ML Providers)
  2. File: tests/integration/test_component_workflows.py::TestRSSToMetadataWorkflow
  3. Validates: RSS → Parse → Download/Transcribe → NER → Local Summarization → Metadata → Files
  4. Uses: RSS feed with audio URL, mocked Whisper, mocked NER, mocked local summarization

  5. This is the complete critical path with ML providers

  6. test_full_workflow_with_openai_providers ⭐ ESSENTIAL (OpenAI Providers)

  7. File: tests/integration/test_component_workflows.py::TestRSSToMetadataWorkflow

  8. Validates: RSS → Parse → Download/Transcribe → OpenAI Speaker Detection → OpenAI Summarization → Metadata → Files

  9. Uses: RSS feed with audio URL, mocked OpenAI transcription, speaker detection, and summarization

  10. This is the complete critical path with OpenAI providers

  11. File: tests/integration/test_component_workflows.py::TestRSSToMetadataWorkflow

  12. Validates: RSS → Parse → Download Transcript → Metadata → Files

  13. Uses: RSS feed with transcript URL

  14. test_rss_to_transcription_workflow (Path 2: Transcription)

  15. File: tests/integration/test_component_workflows.py::TestRSSToMetadataWorkflow
  16. Validates: RSS → Parse → Download Audio → Whisper Transcription → Metadata → Files

  17. Uses: RSS feed without transcript URL (has audio URL)

  18. test_episode_processor_audio_download_and_transcription

  19. File: tests/integration/test_component_workflows.py::TestRSSToMetadataWorkflow
  20. Validates: Episode processor functions for audio download and transcription

  21. Uses: Mocked Whisper transcription

  22. test_speaker_detection_in_transcription_workflow

  23. File: tests/integration/test_component_workflows.py::TestRSSToMetadataWorkflow
  24. Validates: RSS → Parse → Download Audio → Transcribe → NER → Metadata → Files
  25. Uses: Mocked Whisper and NER
  26. Note: Covers NER but not summarization (use test_full_workflow_with_ner_and_summarization for complete path)

Execution Time: ~8-9 seconds

E2E-Fast Tests ✅

3 tests covering critical path:

All E2E tests use transcribe_missing=True, so they validate Path 2 (transcription):

  1. test_cli_basic_transcript_download (CLI)
  2. File: tests/e2e/test_basic_e2e.py::TestBasicCLIE2E
  3. Validates: CLI transcription end-to-end

  4. Uses: --transcribe-missing

  5. test_library_api_basic_pipeline (Library API)

  6. File: tests/e2e/test_basic_e2e.py::TestBasicLibraryAPIE2E
  7. Validates: run_pipeline(config) transcription end-to-end

  8. Uses: transcribe_missing=True

  9. test_service_api_basic_run (Service API)

  10. File: tests/e2e/test_basic_e2e.py::TestBasicServiceAPIE2E
  11. Validates: service.run(config) transcription end-to-end
  12. Uses: transcribe_missing=True

Execution Time: ~30-45 seconds (with minimal audio file)

Test Coverage Matrix

Scenario Integration-Fast E2E-Fast
Transcript URL exists → Download ✅ test_rss_to_metadata_generation ❌ (not needed, covered in integration)
No transcript URL → Transcribe ✅ test_rss_to_transcription_workflow ✅ All 3 E2E tests
Episode processor transcription ✅ test_episode_processor_audio_download_and_transcription ❌ (component-level, integration is sufficient)

Using This Guide

When Writing New Tests

  1. Check if it's critical path: Is this part of the critical path? (RSS → Parse → Download/Transcribe → Speaker Detection → Summarization → Metadata → Files)

  2. Choose test level: Unit (function), Integration (components), or E2E (workflow)?

  3. Prioritize: Critical path tests should be fast and run frequently

When Reviewing Test Coverage

  1. Verify critical path coverage: Do we have tests for all critical path steps?

  2. Check all three levels: Unit, Integration, and E2E tests for critical path?

  3. Ensure fast feedback: Critical path tests should run quickly (with mocks)

When Debugging Failures

  1. Check critical path first: If critical path tests fail, fix those immediately
  2. Verify all entry points: CLI, Library API, and Service API should all work
  3. Test both paths: Transcript download and transcription should both work

Summary

Key Principles:

  1. Critical path = Core value: RSS → Parse → Download/Transcribe → NER → Summarization → Metadata → Files

  2. Test at all levels: Unit, Integration, and E2E tests for critical path

  3. Prioritize fast feedback: Critical path tests should run quickly (with mocks)

  4. Cover all entry points: CLI, Library API, and Service API must all work

  5. Test both paths: Transcript download and transcription are both critical

Remember: If the critical path doesn't work, nothing else matters. Focus your testing efforts on ensuring the critical path is solid, then expand to extended features.

References