WhisperSpeech
Open-source text-to-speech system built by inverting Whisper.
Open-source text-to-speech system built by inverting OpenAI Whisper
WhisperSpeech is an open-source text-to-speech system that inverts OpenAI Whisper, aiming to match Stable Diffusion's impact for audio. It suits developers and researchers building custom voice generation pipelines for smart cities and public sector applications.
What it is
It uses a two-stage token-based pipeline combining OpenAI Whisper for semantic tokens, EnCodec for waveform tokenisation, and Vocos for neural vocoding. It supports fast inference via torch.compile and KV-caching, enabling real-time performance on consumer GPUs.
Who it's for
It suits developers and researchers needing hackable, commercially safe text-to-speech models with voice cloning capabilities. It does not suit users looking for simple plug-and-play end-user applications without technical setup.
Key features
OpenAI Whisper Inversion
Uses Whisper to map transcriptions to semantic tokens for text-to-speech.
Voice Cloning Support
Enables one-click voice cloning from reference audio samples.
Fast GPU Inference
Achieves 12x faster-than-real-time performance on an RTX 4090 using torch.compile and KV-caching.
Multilingual Code-Switching
Supports mixing multiple languages seamlessly within a single sentence.
Up and running in 3 steps
Our honest verdict
What's great
- Trained only on properly licensed data.
- Includes code-switching capabilities across multiple languages.
- Achieves high execution speeds on consumer hardware like the RTX 4090.
- Employs an audio generation pipeline comparable to AudioLM and SPEAR TTS.
Where it falls short
- Requires significant technical skill with PyTorch and Jupyter Notebooks.
- Running high-performance generation requires dedicated consumer or enterprise GPUs.
- Final multilingual models are still undergoing active development and roadmap completion.
Pricing & license
The software is free under its open-source licences, with no paid hosted options mentioned in the README. Your only costs are compute resources required to run the models.
Open remote AI-training roles
767 open remote AI-training roles, updated November 14, 2026; most pay $20–$175 an hour. Zekai lists them free; the application goes to the employer’s platform.
Alternatives
Use it, or have us set it up
FAQ
What data were the models trained on?
The models are trained exclusively on properly licensed data, starting with the English LibreLight release.
Can I perform voice cloning with WhisperSpeech?
Yes, the system supports one-click voice cloning from short audio reference files.
What hardware is required for fast generation?
The developers achieved 12x real-time performance using a consumer RTX 4090 GPU with torch.compile enabled.
