Voice and cloning

Each sentence is dubbed with the voice of whoever says it. vt/dub.py (Dubber) takes the person assigned by diarization, looks up their reference voice and asks the voice process for the sentence in Spanish with that voice:

1. Choose the reference: the person’s own, that of another person you chose in “Personas” (People), or the most similar one if they don’t have one yet.
2. Compute the available space and the duration requested from the engine.
3. Synthesize with OmniVoice (16 steps), cleaning the reference first if it is noisy.
4. Trim silences and, if it didn’t fit, compress with WSOLA.
5. Match the loudness to that of the original voice and save the WAV.

Voice references

A person’s reference is their best original sentences, complete, joined with 0.15 s of silence, along with their transcript. It builds itself as the video advances (SpeakerTracker.add_reference in vt/speakers.py):

• Only sentences of 1.6 s or more contribute (short ones give unreliable voiceprints) and with at least 1 s excluding the silences at the edges.
• Each sentence goes in whole, up to 8 s (MAX_CLIP_S). A sentence of 8 to 12 s (MAX_FALLBACK_S) is used only as a last resort, if the person has no other, and the first sentence of normal length replaces it.
• Each sentence is scored by length and cleanliness: min(duration, 6) × (SNR / 30), with the cleanliness factor between 0.2 and 1. The best ones are kept until they add up to 9 s.
• With the reference already at 75%, a new sentence only gets in if it is 1.3 times better than the worst: each change forces the engine to recompute the voice (1 to 2 s of GPU).

References are stored in <library>/voices/<video>/spk<person>_v<version>.wav, with voices.json (file, version, text and score). Each version gets a new name (the engine caches the computed voice by file path and date, so it never reuses an old one). The current and the previous one are kept, since the voice process may be reading it at that moment; older ones are deleted.

Why up to 8 s and no trimming

A reference is never trimmed

The engine needs the reference’s text to match its audio exactly. If a sentence is cut in half and the text is guessed, the engine “continues” the reference and the generated voice carries over the leftover words: “and so” or “stands” could be heard in every dubbed sentence. That is why only complete sentences are used.

Length has a direct cost: with Whisper occupying the GPU, a 15 s reference made cloning take 61 s per sentence; with 9 s, 4 s. OmniVoice recommends references of 3 to 10 s.

Which voice each sentence uses

Case	Reference
The person has a reference	Their own.
In “Personas” you chose another person’s voice from the video	That person’s (voice_map in the transcript).
You chose a person saved in the library	Their WAV in <library>/people/ (voice_map with the p: prefix).
The person doesn’t have a reference yet	That of the person with a reference whose voiceprint is most similar.

Synthesis

The voice process loads k2-fsa/OmniVoice in float16 and generates with 16 steps and guidance_scale 2.0, passing it the desired duration. The computed voice of each reference is cached (the 8 most recent).

The first cloning takes 25 to 50 s when Whisper already occupies the GPU; later ones, about 2 s. That cost is paid when loading the engine, with a short test sentence (OmniVoice.warm), and not on the video’s first sentence. It repeats every time the model returns from RAM to the GPU.

Cleaning with DeepFilterNet

With clean_ref="auto" (what Dubber uses), the voice process estimates the reference’s signal-to-noise ratio. If it is below 20 dB, it cleans it with DeepFilterNet3 (vt/enhance_df.py, maximum attenuation of 30 dB) and clones from the clean copy spk<person>_v<version>_df.wav. Clean references are left alone: any processing takes a little identity away from the voice.

Measured with tools/eval_tts.py: with pink noise at 5 dB, the clone’s voice similarity rises from 0.45 to 0.59 and the pitch stops drifting.

If DeepFilterNet doesn’t load, the reference is cloned as is. enhance_df.py substitutes a module that torchaudio 2.9 removed and that DeepFilterNet still imports. The custom spectral gate in vt/audiofx.py (denoise) exists, but is disabled by default for the same reason.

Duration

Spanish tends to be longer than English. So that each sentence fits without being distorted, the duration is requested from the engine and WSOLA stays as a fallback:

Step	Rule (vt/dub.py)
Available space	Up to the start of the next sentence, minus 0.08 s (at least 0.4 s). If there is no next sentence yet, the sentence’s duration plus 0.3 s.
Requested duration	That of the original sentence + 5%, or the one given by a pace of 20 characters per second if greater; between 0.5 s and the available space.
Trimming	Silences at the edges, leaving 0.03 s.
Compression	If the audio exceeds the space by more than 2%, WSOLA compresses it up to 1.25 times.

time_stretch (vt/audiofx.py) is a custom WSOLA: 30 ms Hann windows with a 15 ms hop and a ±10 ms search by correlation to continue the waveform. It changes the duration without changing the pitch; tests/test_core.py checks it with a 150 Hz tone compressed 1.3 times.

Loudness

The dubbed voice sounds at the same level as the original:

• From the original sentence, the loudness of the voiced parts is measured (active_rms_db: 30 ms frames, ignoring those more than 30 dB below the loudest). It is stored in the sentence as orig_db.
• The generated voice is brought to that level, bounded between −26 and −14 dB (−20 dB if there is no measurement), with a peak limiter at 0.97.
• An 8 ms fade is applied at each edge.

The result is saved in <library>/audio/<video>/<sentence>.wav (16-bit, mono) and the sentence records dub: file, duration, compression applied, GPU seconds, engine, person and the text that was dubbed.

Re-dubbing

A sentence is dubbed again when its dub no longer matches what the sentence is now:

Change	Why
The translation changed (glossary, another translator)	dub.text differs from translation.
The voice engine changed (--voice)	dub.engine differs.
The sentence changed person (diarization correction)	dub.voice differs from speaker.
In “Personas” you changed someone’s voice	The dub of their sentences is deleted and the video enters the queue.

The queue does it on its own as it passes through those sentences, starting with what you’re about to hear (see Pipeline). Improving a reference doesn’t redo sentences already dubbed: only the upcoming ones use the new version.

Repair old references

Earlier versions trimmed references to 8 s by guessing the text. vt/repair_refs.py rebuilds them:

python -m vt.repair_refs --check   # only lists videos with suspicious references
python -m vt.repair_refs           # repairs them (with the server stopped)

A reference is suspicious if its text doesn’t end in a closing mark (., ?, !, …, », quotes or parentheses). For each affected person it builds a new one from their complete sentences of 1.5 to 8 s (those that don’t close are worth 0.6 times less), up to 9 s; if they have none, it uses a whole sentence of 8 to 12 s, and if there is none either, the person is left without a reference and uses the most similar voice. Then it deletes all of the video’s voice and puts it in the queue to be dubbed again.

Note

repair_refs only checks the Principal library (data/): it reads data/library.json, data/voices/ and data/transcripts/.

Choosing the engine

Measured with tools/eval_tts.py on an RTX 4060: 3 dialogues, 9 people, 18 sentences.

Engine	Voice similarity	Identity	Word error	RTF
OmniVoice (16 steps, fixed duration)	0.78	100%	2.2%	0.27
Chatterbox Multilingual	0.64	100%	17.2% (cuts sentences)	0.83
VoxCPM2	0.02	28%	99% (hallucinates)	—

For reference, two recordings of the same person give between 0.6 and 0.7 similarity, and different people 0.2 or less. With noise in the video, OmniVoice gives 0.76 with music at 10 dB, 0.66 with café noise at 10 dB and 0.59 with pink noise at 5 dB.

Each engine lives in its own environment (.venvs/omnivoice, .venvs/chatterbox, .venvs/voxcpm) and is chosen at startup with --voice. With --voice none, nothing is dubbed. Memory usage and load times are in GPU and idle.