corentinj/real-time-voice-cloning
Clone a voice in 5 seconds to generate arbitrary speech in real-time
How real-time-voice-cloning is put together
This repository implements SV2TTS (Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis) as a three-stage deep learning pipeline: a GE2E speaker encoder that produces voice embeddings from short audio clips, a Tacotron-based synthesizer that generates mel spectrograms conditioned on those embeddings and input text, and a WaveRNN vocoder that converts spectrograms to audio in real time. It is written entirely in Python with PyTorch as the deep learning framework, packaged via uv/pyproject.toml, and pinned to Python >=3.9,<3.10. Users interact through two entry points: demo_toolbox.py, a PyQt5 GUI application, and demo_cli.py, a command-line interface. Data flows from audio files/datasets (LibriSpeech, LibriTTS, VoxCeleb, LJSpeech, VCTK) through preprocessing scripts into training pipelines, while pretrained models are fetched at runtime from Hugging Face Hub.
Languages
Frameworks
Infrastructure
Major components
encoder
Implements the GE2E speaker verification encoder: audio processing, LSTM embedding model, data loading objects, preprocessing, training loop, and inference to produce 256-dim voice embeddings.
synthesizer
Implements the Tacotron-2 style synthesizer: text cleaning/symbolization utilities, dataset handling, training, and inference generating mel spectrograms from text plus speaker embeddings.
vocoder
Implements the WaveRNN vocoder (fatchord and deepmind variants): distribution modeling, dataset handling, training, and real-time generation of waveforms from mel spectrograms.
toolbox
Provides the interactive PyQt5 GUI (toolbox/ui.py) and utterance management (toolbox/utterance.py) tying together encoder, synthesizer, and vocoder for live voice cloning.
utils
Shared helpers: CLI argument parsing (argutils.py), pretrained model definitions/download (default_models.py), logmmse noise reduction (logmmse.py), and profiling (profiler.py).
training entry-point scripts
Top-level scripts (encoder_train.py, synthesizer_train.py, vocoder_train.py and their preprocess counterparts) drive per-stage dataset preprocessing and model training from the command line.
demo entry points
demo_toolbox.py launches the GUI toolbox and demo_cli.py runs the full clone-speech pipeline non-interactively from the command line.
No digest this week
1 weeksWeek of
Dependencies and code review
Dependency advisories
Security Watch
4 vulnerable packages (33 advisories, 1 actively exploited) across 66 scanned dependencies.
pillow 8.4.0
18 advisories, actively exploited. Fixed in 10.2.0.
How it can be abusedAn attacker would need to trick the program into opening a maliciously crafted image file that runs code or crashes it when loaded. This project only handles audio files and text, so images are never processed from untrusted sources, leaving ordinary users of the demo tools essentially unaffected.
- criticalGHSA-3f63-hfp8-52jqArbitrary Code Execution in Pillow
- highGHSA-44wm-f244-xhp3Pillow buffer overflow vulnerability
- highGHSA-45hq-cxwh-f6vcPillow `BdfFontFile`: `Image.new()` called without `_decompression_bomb_check()` — bomb protection bypass via font loading
- highGHSA-5x94-69rx-g8h2Pillow: `FontFile.compile()`: `Image.new()` called without `_decompression_bomb_check()`
- highGHSA-62p4-gmf7-7g93Pillow: Out-of-bounds read via attacker-controlled row stride on Pillow's mmap path (McIdas AREA files)
- highGHSA-6r8x-57c9-28j4Pillow: Heap out-of-bounds write `Image.paste()` / `Image.crop()` via signed coordinate overflow
- highGHSA-8ghj-p4vj-mr35Pillow Denial of Service vulnerability
- highGHSA-8v84-f9pq-wr9xPillow `PcfFontFile._load_bitmaps()`: `Image.frombytes()` called without `_decompression_bomb_check()` — bomb protection bypass via PCF font loading
- criticalGHSA-8vj2-vxx3-667wArbitrary expression injection in Pillow
- highGHSA-9hw9-ch79-4vh6Pillow: Controlled heap out-of-bounds write in Pillow `ImageCmsTransform.apply()` via output mode mismatch
- highGHSA-9j59-75qj-795wPath traversal in Pillow
- highGHSA-j7hp-h8jx-5pprexploitedlibwebp: OOB write in BuildHuffmanTable
- highGHSA-jjj6-mw9f-p565Pillow: Decompression Bomb DoS via PdfParser.PdfStream.decode()
- highGHSA-m2vv-5vj5-2hm7Pillow vulnerable to Data Amplification attack.
- highGHSA-phj9-mv4w-65pmPillow `GdImageFile._open()`: image dimensions accepted without `_decompression_bomb_check()`
- highGHSA-vjc4-5qp5-m44jPillow JPEG2000 tiled decode retains a growing scratch buffer and can be used for denial of service
- highGHSA-xj96-63gp-2gmrPillow: Heap out-of-bounds write in `ImageFilter.RankFilter` via integer overflow in `ImagingExpand`
- unknownPYSEC-2026-1794exploitedlibwebp: OOB write in BuildHuffmanTable
torch 1.10.2
4 advisories. Fixed in 1.13.1.
How it can be abusedThis project downloads pretrained PyTorch models from the internet and loads them with torch.load, which can run hidden code inside a tampered checkpoint file. An attacker who poisons a model file, or tricks a user into loading one, gets full control of the machine of anyone running the toolbox GUI or CLI.
- criticalGHSA-47fc-vmwq-366vPyTorch vulnerable to arbitrary code execution
- criticalGHSA-53q9-r3pm-6pq6PyTorch: `torch.load` with `weights_only=True` leads to remote code execution
- highGHSA-5pcm-hx3q-hm94PyTorch heap buffer overflow vulnerability
- highGHSA-pg7h-5qx3-wjr3Pytorch use-after-free vulnerability
tornado 6.5.2
6 advisories. Fixed in 6.5.6.
How it can be abusedAn attacker could exploit these flaws mainly if this project ran a Tornado-based web server or fetched URLs through its HTTP client, for example by sending crafted requests that exhaust memory or by tricking a client into leaking login headers to another site. This repo is a desktop GUI and command line tool that only downloads pretrained models from Hugging Face, so it does not run a public server and ordinary users are not realistically exposed.
- highGHSA-3x9g-8vmp-wqvfTornado: Authorization header forwarded across cross-origin redirects in SimpleAsyncHTTPClient
- highGHSA-c98p-7wgm-6p64Tornado: Quadratic DoS via Repeated Header Coalescing
- highGHSA-fqwm-6jpj-5wxcTornado has cookie attribute injection via .RequestHandler.set_cookie
- highGHSA-jhmp-mqwm-3gq8Tornado: Quadratic DoS via Crafted Multipart Parameters
- highGHSA-mgf9-4vpg-hj56tornado AsyncHTTPClient accumulates decompressed chunks without size limit (gzip bomb)
- highGHSA-qjxf-f2mg-c6mcTornado is vulnerable to DoS due to too many multipart parts
urllib3 1.26.7
5 advisories. Fixed in 2.6.0.
How it can be abusedThis project downloads pretrained models from the internet at runtime using Python's HTTP stack, which relies on this library. A malicious or compromised download server could send endless compressed data to freeze or crash the toolbox, or trick the client into following redirects that leak cookies or login headers to other sites, affecting anyone running demo_toolbox.py or demo_cli.py while fetching models.
- highGHSA-2xpw-w6gg-jr37urllib3 streaming API improperly handles highly compressed data
- highGHSA-38jv-5279-wg99Decompression-bomb safeguards bypassed when following HTTP redirects (streaming API)
- highGHSA-gm62-xv2j-4w53urllib3 allows an unbounded number of links in the decompression chain
- highGHSA-qccp-gfcp-xxvcurllib3: Sensitive headers forwarded across origins in proxied low-level redirects
- highGHSA-v845-jxx5-vc9f`Cookie` HTTP header isn't stripped on cross-origin redirects
Code review
No concrete code-level issues confirmed.
Get this report every week for your repos.
GitZoid learns each repo, reports what changed, and flags what needs attention. One flat price for the whole team.
$19 a month, flat · First 10 outputs free · No card required