MuseTalk
Real-time audio-driven lip-sync model that re-animates the mouth of an unseen face in video, with training code.
Match a face's mouth to new audio at 30fps+ on a V100-class GPU.
MuseTalk is a lip-syncing model from Tencent Music Entertainment that changes the mouth region of a face in a video or image to match a new audio track. It suits developers and researchers with an NVIDIA GPU who want dubbing in several languages.
What it is
It works in the latent space of a VAE, inpainting a 256x256 face region in a single step, so it is not a diffusion model. The README says it supports Chinese, English and Japanese audio, runs at 30fps+ on an NVIDIA Tesla V100, and that version 1.5 adds clarity, identity consistency and sync improvements.
Who it's for
Suits developers building dubbing or virtual-human pipelines and researchers who want training code. It does not suit non-technical editors: setup involves conda, PyTorch 2.0.1, the OpenMMLab stack and many downloaded weights.
Key features
Real-time inference
A separate realtime script targets 30fps+ on an NVIDIA Tesla V100.
Multilingual audio
The README says it handles audio in languages such as Chinese, English and Japanese.
Adjustable face centre
You can change the centre point of the face region, which the README says strongly affects results.
Version 1.5 model
Adds perceptual, GAN and sync losses with two-stage training and spatio-temporal sampling.
Training code released
Scripts and configs let you train your own MuseTalk model.
ComfyUI integration
A third-party ComfyUI node is linked, which the project says it does not maintain.
Up and running in 3 steps
Our honest verdict
What's great
- Code and trained models are MIT, and the README states the model can be used for any purpose, including commercially.
- Inference and training code for version 1.5 are both published, which most lip-sync releases do not offer.
- A realtime inference path exists, not just batch rendering.
- It was trained with sync loss and ships with a SyncNet weight, so it targets mouth accuracy.
Where it falls short
- It needs an NVIDIA GPU with CUDA; the README recommends Python 3.10, CUDA 11.7 and PyTorch 2.0.1.
- Setup is long: MMLab packages (mmcv, mmdet, mmpose), FFmpeg configuration and weights from seven separate sources.
- Training is memory-hungry; the README lists stage 2 at roughly 54 to 85 GB per GPU depending on batch size.
- Other models it uses (whisper, dwpose, S3FD, the VAE) keep their own licences, and the test data is research use only.
Pricing & license
MuseTalk is MIT-licensed and free to run yourself. The README shows a Hugging Face Space demo but no paid plan. Your cost is a CUDA GPU machine and the setup time.
Open remote AI-training roles
768 open remote AI-training roles, updated November 14, 2026; most pay $20–$175 an hour. Zekai lists them free; the application goes to the employer’s platform.
Alternatives
These reviewed AI tools do the same job instantly — from our scored directory.
Use it, or have us set it up
FAQ
Can I use MuseTalk commercially?
The README says the code is MIT and the trained model is available for any purpose, even commercially. Other open models it depends on follow their own licences.
What hardware do I need?
The README quotes 30fps+ on an NVIDIA Tesla V100 and lists CUDA 11.7 and PyTorch 2.0.1 in setup, so plan for a CUDA GPU.
Is it a diffusion model?
No. The README says it resembles Stable Diffusion's UNet but inpaints in latent space in a single step.
