Skip to content

Model bake-off — overnight autonomous run (2026-08-05)

Autonomous continuation while operator sleeps. Goal: map the full cost-vs-value frontier for the podcast enrichment pipeline (summary + insights + KG topics) across model families, so we can pick the model that beats Gemini for less money at 100-ep scale.

Method (identical for every model — model is the ONLY variable)

  • Harness: --pipeline-stage relabel_only --reprocess-existing-only over the SAME 9 diverse episodes (one per feed) copied from prod-v2.4-relabel-fixed. Reuses frozen 2.4 ASR + diarization; re-runs the whole LLM surface (naming → cleaning → summary → GI → KG).
  • Config: one frozen control config; only the model alias/provider changes. Robust settings (GI chunk 20k, timeout 600s, insights cap 8, dedup 0.72, value-gate tier 3, summary cap 2048).
  • Judge: Claude Opus 4.8 (dedicated AUTORESEARCH_JUDGE_ANTHROPIC_API_KEY), blind A/B, length-controlled + position-controlled, scoring top-12 ranked insights (production view) — summary / insights / topics, 1–10 + winner. Baseline contestant = Gemini 2.5-flash-lite (v2.4).
  • Cost: LiteLLM SpendLogs delta / 9 eps (actual, all calls).

Leaderboard (fair top-12; vs Gemini-2.5-flash-lite)

Model $/ep Summary Insights Topics Overall Notes
qwen3.7-flash 0.0045 8.44 8.22 7.78 9-0 VALUE WINNER — cheaper than flash-lite, near-deepseek quality
deepseek-v4-flash-0731 0.0092 8.44 8.44 8.22 9-0 QUALITY CHAMPION
gemini-2.5-flash 0.0491 8.22 8.00 7.89 9-0 beaten by qwen+deepseek on quality AND 5–10x pricier
glm-4.7-flash 0.0182 8.29 7.86 7.57 7-0* decent quality but PRICIEST + PATHOLOGICALLY SLOW (56min, >10min GI calls); *7-ep
Gemini-2.5-flash-lite 0.0047 (baseline) — — — cheapest Gemini (v2.4 baseline)
gemini-2.5-pro 0.2115 8.56 7.89 7.89 8-1 ✅ GI-bug FIXED (c7877ffa: bounded thinking 1024 + max_output_tokens headroom). Re-run: 0 stub-fallback across 9 eps. BEST summary of ALL models (8.56), but insights/topics only mid-pack; EXACT cost $1.904/9ep — ~23–47x the flash models + pathologically SLOW (~2h40m for 9 eps). Dominated.
deepseek-v4-pro ~0.03+ 8.44 7.11 6.33 7-2 good summary but WEAK topics (loses to Gemini) + SLOW (36min on WSJ, hung) + pricey (9-ep)
gpt-4.1-mini (openai) ~0.020 7.67 6.89 7.44 6-3 external ref: beats Gemini-lite but BELOW qwen/deepseek on every dimension + pricier
kimi-k2.6 ~0.044+ 8.50 8.38 7.88 8-0* BEST insights (8.38) but priciest output ($2.48/Mtok) + slow (~50min); *8-ep. Dominated by deepseek-flash (= quality, 5-10x cheaper, faster)
glm-5.2 (full) ~0.035 8.44 8.22 8.11 8-1 full GLM FIXES glm-flash's weak topics (8.11 vs 6.86) — matches deepseek-flash quality, but 4x pricier + slow (~58min, timeouts)
gpt-4.1-mini (openai) running

Pro/reasoning tier verdict (gempro, deepseek-v4-pro, glm-4.7-flash, kimi): all pathologically SLOW (10–14 min GI calls, repeated 600s timeouts, occasional keepalive-hangs) and far pricier — impractical for this pipeline regardless of quality, and their quality does NOT clearly beat the flash models (weak topics or broken GI). The FLASH models win the practical frontier.

Interim read: the two cheap Chinese flash models (qwen3.7-flash, deepseek-flash) BEAT mid-tier gemini-2.5-flash on quality while costing 5–10x less. gemini-2.5-flash beats only flash-lite. glm is out (slow + pricey). qwen3.7-flash is the standout value; deepseek-flash the quality lead.

Queue (sequential — avoid machine thrash; slow runs finish, "slow" is a recorded finding)

  1. glm-4.7-flash iter-1 (finishing) → judge → record.
  2. gemini-2.5-flash → judge. 3. gemini-2.5-pro → judge.
  3. deepseek-v4-pro → judge. 5. kimi-k2.6 → judge. 6. glm-5.2 → judge.
  4. gpt (openai, tbd model) → judge.
  5. Compile the frontier + a recommendation for the 100-ep scale run.

Rules held to

  • Never stop/kill a run without operator approval — slow runs run to completion.
  • Commit each profile + record results as they land.
  • Flag any real decision (which model(s) advance to 100-ep) for the operator, don't decide it.

FRONTIER + RECOMMENDATION (for operator review)

The two cheap Chinese FLASH models win outright. They beat every Gemini tier, gpt-4.1-mini, and every pro/reasoning model on quality-per-dollar — and they're the only ones that are also fast.

  • qwen3.7-flash — VALUE WINNER. $0.0045/ep (cheaper than the flash-lite baseline itself), beats Gemini-lite 9-0, quality 8.44 / 8.22 / 7.78. Best cost-quality point on the whole frontier.
  • deepseek-v4-flash-0731 — QUALITY CHAMPION. $0.0092/ep, beats Gemini-lite 9-0, best-rounded 8.44 / 8.44 / 8.22. ~2x qwen's cost for a small topic/insight edge.

Everything else is dominated: - All Gemini tiers: flash-lite is the weak baseline; gemini-2.5-flash is beaten on quality AND 10x pricier; gemini-2.5-pro (GI-bug now FIXED, c7877ffa) has the BEST summary of any model (8.56) and beats the baseline 8-1, but its insights/topics are only mid-pack, it costs $0.21/ep (~23–47x the flash winners), and it is pathologically slow (~2h40m for 9 eps) — dominated. - gpt-4.1-mini (external ref): below qwen/deepseek on every dimension + pricier (~$0.02/ep). - Pro/reasoning tier (deepseek-v4-pro, kimi-k2.6, glm-4.7-flash, glm-5.2): all pathologically SLOW (10–36 min per long episode, 600s timeouts, keepalive-hangs) and far pricier; their quality is at best comparable to the flash models, often with WEAK topics. Impractical for our pipeline.

Recommendation for the 100-ep scale validation

Run qwen3.7-flash (primary — best value) and deepseek-v4-flash-0731 (quality reference) at 100-ep scale; both are cheap and fast enough to run together (~$0.5–1 combined for 100 eps). Confirm the win holds at scale, then default the pipeline to qwen3.7-flash (or deepseek-flash if the topic edge matters for downstream KG).

Decisions flagged for operator

  1. Which model(s) advance to the 100-ep run — recommend qwen3.7-flash + deepseek-flash.
  2. gemini-2.5-pro GI bug: FIXED (c7877ffa) + re-run — fair number now on the board (best summary 8.56, beats baseline 8-1) but dominated on cost ($0.21/ep) and speed. No open action; kept as a reference row.
  3. Cost caveat: pro/reasoning-tier per-ep costs are token-estimates (parallel runs shared the LiteLLM key spend); flash-model costs are exact.

Results log

(updated as runs complete — see git commits tune(bakeoff): … / docs(wip): … and the table)