canary

A 78-minute deposition went through WhisperX and came back missing 57 minutes, with no error and no warning. I built the pipeline that makes that failure mode structurally impossible, split across two repos by how often each half actually needs to change.

Overview

The transcript from WhisperX looked fine: clean sentences, reasonable pacing, nothing that would make a reviewer suspicious. It was also missing three quarters of the recording. WhisperX runs a voice-activity detector ahead of transcription and only feeds the model audio it classifies as speech; on a real recording (phone compression, overlapping speakers, someone shuffling papers) that classifier is wrong constantly, and everything it gets wrong is simply gone. No gap marker, no warning, no non-zero exit code. The transcript reads clean precisely because the parts it dropped are the parts you never see.

The fix is structural, not a tuning knob: nothing gets to decide what counts as speech before transcription runs. Audio is walked in fixed, heavily overlapping windows (30 seconds wide, 20 seconds of target overlap) so every interior moment is covered by two or three independent passes; silence detection only chooses where a window boundary falls, never whether a window exists. Two independent ASR models, Parakeet-TDT and Canary-1b-flash, transcribe the same audio rather than one. Where they agree, that is high-confidence content. Where they disagree, the disagreement is surfaced for review instead of one model silently overruling the other. Diarization is kept out of the decision entirely: pyannote labels who is speaking, but it never gets a vote on whether a word gets transcribed. A word outside every detected speaker turn still ships, tagged unknown instead of deleted.

The whole thing runs on rented Vast.ai GPUs, provisioned, transcribed, downloaded, and torn down end to end. A full 30-minute recording costs single-digit cents and finishes in under 10 minutes with the current image.

Technical

Two repos, split by rate of change rather than by convenience. canary-pipeline is everything that runs on my own machine: audio prep, provisioning and monitoring the rented GPU, pulling results back, local diarization tuning, and the test suite; changes here are application logic and ship instantly. canary-asr is the Docker image that runs on the rented GPU itself: NeMo, pyannote, a version-pinned CUDA-aligned PyTorch, and both models’ weights baked in at build time; changes here mean a new image and a roughly 20-minute CI build. Splitting them means a config tweak in the orchestration logic never triggers a rebuild of a multi-gigabyte image, and a dependency bump in the ASR stack never touches the Python that runs locally.

The image itself replaced NVIDIA’s official NeMo container, which is 25 to 30 GB and, on a rented instance torn down after every job, taxes every single rental with a 20 to 30 minute pull; on a mediocre host the pull sometimes never finished at all. The custom image is around 13 GB, most of it model weights rather than framework bloat, and both ASR models’ weights are baked in at build time rather than pulled from HuggingFace’s throttled anonymous-access path on every run, which used to cost 10 to 15 minutes of dead time before a single second of transcription happened.

Several of the guardrails exist because a specific incident put them there. Early on, the transcription loop checkpointed every five batches but had no final write once the loop finished, so a run silently lost five minutes and forty two seconds of audio with a clean exit code; the fix is a checkpoint on every batch, an explicit final write, and a hard guard that writes a critical warning if the last transcribed segment ends more than 30 seconds before the audio does. The overlapping windows that make the pipeline exhaustive also duplicate text, up to 56 percent inflation in early testing; the fix is timestamp-ownership partitioning, where each window owns the exact time range closest to its center and a word is kept only if it falls inside its own window’s owned range, correct by construction regardless of overlap depth. On a medical recording, one model transcribed a term with a “hyper” prefix at a timestamp where the other model transcribed the same term with “hypo,” a disagreement that inverts clinical meaning rather than just garbling a word; a cross-model check now flags exactly that polarity-prefix pattern into its own review file instead of trusting either transcript by default. On the infrastructure side, roughly half of rented GPU hosts turned out to be bad, either dying mid-pull or stuck in a Docker retry loop with zero forward progress; a stuck-host detector now distinguishes genuinely slow from confirmed dead and automatically destroys and re-provisions on a fresh host, live-validated against real Vast.ai infrastructure, including at least one real bad host it caught and correctly rerolled.

Every run produces machine-readable diagnostics alongside the transcript: a coverage floor and trailing-gap warnings file, per-model coverage percentages, cross-model polarity-inversion flags, pattern-matched hallucination detection, and per-speaker summaries. warnings.json is the first thing worth reading; an empty array means the two hard guards did not trip. The pipeline carries 118 automated tests across 10 files, roughly 3 seconds to run, with no GPU or NeMo dependency, including a dedicated regression test for each incident above.

Why You Should Care

A transcript that reads cleanly tells you nothing about whether it is complete. WhisperX’s output read cleanly too, right up until someone checked it against the source audio and found 57 missing minutes. A lot of exhaustive-transcription setups treat that as a tuning problem; this one treats it as a design constraint, which is why the VAD gate is removed rather than adjusted, and why two independent models surface disagreement rather than one model winning by default.

None of the hardening here came off a whiteboard. Every guardrail in the technical section traces to a specific incident on a real recording: a silent tail-loss, duplicate text from overlapping windows, a clinically dangerous polarity inversion, an infrastructure host that looked slow but was actually dead. That is what it costs to make a transcription pipeline trustworthy enough to hand someone a deposition transcript and mean it, and it is also why the diagnostics ship alongside the transcript instead of being trusted implicitly.

Status

Both repos are active and in production use. Live-validated against real Vast.ai infrastructure, not just simulated.

Repos: https://github.com/VolanticSystems/canary-asr and https://github.com/VolanticSystems/canary-pipeline