whisper-diarization
Combines OpenAI Whisper with NeMo and Demucs to transcribe audio and label distinct speakers.
Combine OpenAI Whisper ASR with Nvidia NeMo speaker embeddings for timestamped multi-speaker transcripts.
whisper-diarization is an open-source Python pipeline that transcribes audio and attributes spoken words to specific individuals. Executive assistants and note-takers can use it to convert raw meeting recordings into speaker-separated text without sending data to external APIs.
What it is
The pipeline isolates vocal stems using Demucs, transcribes speech via OpenAI Whisper, and aligns timestamps with ctc-forced-aligner. It then uses Nvidia NeMo MarbleNet for voice activity detection and TitaNet to extract speaker embeddings that assign speaker identities to word-level timestamps.
Who it's for
It suits technical professionals and executive assistants who need automated speaker identification for private meeting transcripts. It does not suit non-technical users looking for a graphical app or teams lacking access to local GPU hardware.
Key features
Vocal Isolation Preprocessing
Extracts vocal tracks from raw audio using Demucs to minimize background noise before running speaker embedding.
CTC Forced Alignment
Aligns Whisper transcription timestamps using ctc-forced-aligner to reduce diarization timing errors.
VAD and Speaker Embeddings
Employs NeMo MarbleNet to remove silence and TitaNet to map individual segments to distinct speaker IDs.
Experimental Parallel Script
Offers a parallel execution script for systems with 10GB or more VRAM to run NeMo and Whisper concurrently.
Up and running in 3 steps
Our honest verdict
What's great
- Combines vocal stem isolation, VAD, and speaker embeddings for precise multi-speaker transcripts.
- Realigns output using punctuation models to adjust for minor time shifts across word boundaries.
- Supports numerical suppression options to transcribe numbers as words for improved alignment.
- Runs entirely on your local infrastructure without third-party service subscriptions.
Where it falls short
- Requires CLI setup, Python 3.10+, FFMPEG, Cython, and system-level build tools.
- Demands significant local VRAM, needing 10GB or more to run parallel processing.
- Cannot separate overlapping speakers speaking simultaneously.
Pricing & license
whisper-diarization is completely free under the BSD-2-Clause license. There are no paid tiers or hosted SaaS options provided by the author, so your only expense is the local hardware or compute time required to run it.
Open remote AI-training roles
873 open remote AI-training roles, updated November 15, 2026; most pay $20–$170 an hour. Zekai lists them free; the application goes to the employer’s platform.
Alternatives
These reviewed AI tools do the same job instantly — from our scored directory.
Use it, or have us set it up
FAQ
Does whisper-diarization handle overlapping speakers?
No, overlapping speech is a known limitation that is not supported in the current version.
Can I run this tool on a CPU?
Yes, you can set the device flag manually, but execution will be substantially slower compared to using CUDA acceleration.
What audio file formats are supported?
It supports any standard audio format that can be processed by FFMPEG, such as WAV, MP3, or M4A.
