whisper_streaming
Real-time streaming transcription and translation for Whisper models.
Turn OpenAI Whisper into a low-latency speech-to-text and translation system.
Whisper-streaming provides real-time speech transcription and translation by building local agreement policy and self-adaptive latency on top of Whisper models. It suits developers and technologists building live audio processing pipelines for multilingual events.
What it is
The tool processes unsegmented long-form speech using multiple backends like faster-whisper, whisper-timestamped, OpenAI API, or Whisper MLX. It implements voice activity detection, chunking policies, and optional buffer trimming strategies to keep latency around 3.3 seconds.
Who it's for
It suits developers integrating real-time transcription into applications or live conference feeds. It does not suit end-users looking for a ready-to-use desktop application without command-line configuration.
Key features
Multiple Backends
Supports faster-whisper, whisper-timestamped, OpenAI API, and Apple Silicon MLX backends.
Voice Activity Detection
Integrates optional VAD and voice activity controllers using PyTorch.
Buffer Trimming
Offers segment or sentence-level buffer trimming options to manage latency and quality.
Real-Time Simulation
Includes scripts to simulate live streaming directly from local 16kHz mono WAV files.
Up and running in 3 steps
Our honest verdict
What's great
- Achieves low latency around 3.3 seconds on unsegmented long-form speech.
- Flexible backend choice including GPU-accelerated and local Apple Silicon options.
- Optional voice activity controllers and detectors reduce processing overhead.
- Built-in support for live translation tasks alongside transcription.
Where it falls short
- Note that the project has been succeeded by SimulStreaming in 2025.
- Requires technical setup of Python environments, audio libraries, and optional GPU drivers.
- Using the OpenAI API backend incurs ongoing per-request API costs that scale with chunk size.
Pricing & license
Free under the MIT license. If you choose to use the OpenAI API backend, you will need to pay OpenAI for API access costs based on your usage.
Open remote AI-training roles
874 open remote AI-training roles, updated November 15, 2026; most pay $20–$170 an hour. Zekai lists them free; the application goes to the employer’s platform.
Alternatives
Use it, or have us set it up
FAQ
What backends are supported?
It supports faster-whisper, whisper-timestamped, OpenAI API, and Whisper MLX for Apple Silicon.
Is a GPU required?
A GPU with CUDA support is recommended for faster-whisper, but you can also use Apple Silicon MLX or the OpenAI API backend.
How does it handle latency?
It uses a local agreement policy with self-adaptive latency and minimum chunk sizing to process audio in real time.
