README.md
13.6 KB · 288 lines · markdown Raw
1 ---
2 base_model: hexgrad/Kokoro-82M
3 base_model_relation: adapter
4 pipeline_tag: text-to-speech
5 license: apache-2.0
6 language:
7 - en
8 datasets:
9 - parler-tts/libritts_r_filtered
10 - facebook/voxpopuli
11 - amphion/Emilia-Dataset
12 tags:
13 - kokoro
14 - voice-cloning
15 - adapter
16 - styletts2
17 - text-to-speech
18 ---
19
20 # kokoro-inno-clone-tuner
21
22 ![banner](https://cdn-uploads.huggingface.co/production/uploads/6444c7a59174daa2f6934a16/_u14iRNVPEU0GA3z7jk48.png)
23
24 ```python
25 pip install inno-kokoro
26 ```
27
28 Zero-shot voice tuner for [Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M).
29
30 Outputs base Kokoro compatible voice packs @ `[510, 1, 256]`. ([write-up](https://remsky.art/writeups/kokoro-fastapi/inno-clone/))
31
32
33 ## Samples
34
35 Same passage for every voice, enrolled from the references. LibriTTS-R speakers are `dev-clean` held out from training.
36
37 | voice | tuned | reference |
38 |---|---|---|
39 | am_libritts1272 | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/libritts_1272_clone.wav"></audio> | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/libritts_1272_ref.wav"></audio> |
40 | af_libritts5338 | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/libritts_5338_clone.wav"></audio> | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/libritts_5338_ref.wav"></audio> |
41 | am_libritts6241 | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/libritts_6241_clone.wav"></audio> | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/libritts_6241_ref.wav"></audio> |
42 | af_ameliaearhart | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/amelia_earhart_clone.wav"></audio> | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/amelia_earhart_ref.wav"></audio> |
43 | bf_janegoodall | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/jane_goodall_clone.wav"></audio> | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/jane_goodall_ref.wav"></audio> |
44 | am_vincentprice | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/vincent_price_clone.wav"></audio> | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/vincent_price_ref.wav"></audio> |
45
46
47
48 ## Usage
49
50 Code available on Github: [remsky/inno-kokoro](https://github.com/remsky/inno-kokoro).
51
52 Integrated into [Kokoro-FastAPI](https://github.com/remsky/Kokoro-FastAPI) (v0.9.0+)
53
54 <details open>
55 <summary>PyTorch, with <a href="https://github.com/hexgrad/kokoro">kokoro</a></summary>
56
57 ```bash
58 pip install "inno-kokoro[torch]"
59 ```
60
61 ```python
62 from inno_kokoro.enroll import Tuner, enroll, read
63 import torch
64 from kokoro import KPipeline
65
66 tuner = Tuner(backend="torch") # caches model from HF on first use
67 pack, _ = enroll(*read("my_ref.wav"), tuner, return_pt=True) # [510, 1, 256]
68 torch.save(pack, "am_me.pt")
69
70 pipe = KPipeline(lang_code="a")
71 wav = next(pipe("Hello from a tuned voice.", voice=pack)).audio
72 ```
73
74 </details>
75
76 <details>
77 <summary>ONNX (no torch), with <a href="https://github.com/thewh1teagle/kokoro-onnx">kokoro-onnx</a></summary>
78
79 ```bash
80 pip install inno-kokoro kokoro-onnx
81 ```
82
83 ```python
84 from inno_kokoro.enroll import Tuner, enroll, read
85 import numpy as np
86 from kokoro_onnx import Kokoro
87
88 tuner = Tuner(backend="onnx") # downloads model.safetensors + model.onnx on first use
89 pack, _ = enroll(*read("my_ref.wav"), tuner) # [510, 1, 256] float32 NumPy
90 np.save("am_me.npy", pack)
91
92 kokoro = Kokoro("kokoro-v1.0.onnx", "voices-v1.0.bin") # from the kokoro-onnx releases
93 wav, sr = kokoro.create("Hello from a tuned voice.", voice=pack, lang="en-us")
94 ```
95
96 </details>
97
98 Notes:
99 - prefix names with stock accent+gender codes for use in other libraries: e.g `af_`, `am_`, `bf_`, `bm_`
100 - pitch ceiling is set automatically
101 - `enroll(..., fmax=180)`: can override when needed (archival sources, etc)
102
103 ### ONNX
104
105 `model.onnx`: the speaker encoder and style head for onnxruntime (fp32, 31.5 MB). Used by `Tuner(backend="onnx")` from inno-kokoro 0.4.0. Same packs as torch, about 2x faster on CPU.
106
107 <details>
108 <summary><b>Graph details</b></summary>
109
110 - input `fbank` [1, frames, 80] float32: kaldi log-mel of the first 30 s at 16 kHz (`inno_kokoro/fbank.py`)
111 - outputs `embedding` [1, 512] (unit, UniSpeech-SAT-sv space) and `style` [1, 256]
112 - opset 20, standard ops only, no DFT or custom ops
113 - matches torch within 5.4e-7 on 540 clips ([parity check](onnx_parity.md))
114 - exported with `export_onnx.py` (inno-kokoro `tools/export_onnx.py` at b442727)
115
116 | reference | onnxruntime | torch |
117 |---|---|---|
118 | 8 s | 100 ms | 165 ms |
119 | 30 s | 410 ms | 890 ms |
120
121 CPU (best of 5 runs)
122
123 </details>
124
125 ### Architecture
126
127 A 5 to 30 s reference clip in, a stock-shaped Kokoro voice pack out. Everything loads from `model.safetensors`, no extra downloads.
128
129 <figure>
130 <img
131 src="https://cdn-uploads.huggingface.co/production/uploads/6444c7a59174daa2f6934a16/XKJQUMieuxKyu5xKu9dc3.png"
132 alt="Inno-Kokoro enrollment architecture"
133 width="95%"
134 />
135 <figcaption>Enrollment Architecture Diagram: reference audio to Kokoro-compatible voice pack.</figcaption>
136 </figure>
137
138 <details>
139 <summary><b>Architecture details</b></summary>
140
141 Enrollment embeds an input audio sample via the baked speaker encoder and maps through `model.safetensors` to build the encoder side; prosody aspects are measured to build the predictor half.
142
143 - **speaker encoder**: a 6.6M-param WeSpeaker ResNet34 distilled from [**UniSpeech-SAT-sv**](https://huggingface.co/microsoft/unispeech-sat-base-plus-sv) embeddings
144 - matches within 0.002 speaker cosine on hold-out dev-clean (380mb fp32 -> 14mb fp16)
145 - **style head**: the decoder (timbre) half of the pack comes from the speaker embedding (512) mapped to Kokoro style (256). Shifts along a learned spectral-tilt direction by the reference's measured tilt.
146 - **prosody blend**: the predictor (prosody) half uses the stock Kokoro packs as a foundation:
147 - Blended by nonnegative least squares to the reference F0 mean, F0 spread and syllable rate.
148 - Packs more than 4 st from the reference's pitch are excluded.
149 - Higher graded packs are weighted as preferred during selection.
150 - **prosody head** (v0.2): a linear adjust on the blend from the reference's F0 mean and spread,
151 - fitted to deltas optimized through the frozen predictor. `enroll(..., head=False)` skips it.
152
153 Enrollment runs in about 0.05 s per second of reference on CPU (a 30 s reference in 1.4 s) and 0.1-0.3 s total on a GPU, after the one-time model load.
154
155 Total model size @ fp16: ~24mb
156
157 </details>
158
159 ## Limits
160
161 - English trained (LibriTTS-R) currently
162 - Some texture/identity is traded to maintain audio quality and stability within Kokoro's range.
163 - Rythym is a best-effort match to the speaker.
164 - Prosody is a convex blend of stock prosody by pitch, spread and pace. Outside of that range, it will only pin to the nearest edge
165 - Reference clip requirements:
166 - 3 second minimum, up to a max of 30s
167 - Reasonably free of audio artifacts
168 - Single speaker
169
170 ## Training and Metrics
171
172 All stages: Kokoro frozen; LibriTTS-R train-clean-100 unless noted.
173
174 Trained via HF Jobs `a10g-small`; under $20 in GPU time:
175
176 <details>
177 <summary>v0.1.0 training notes</summary>
178
179 - Stage 1: mapped speaker embedding to timbre:
180 - Style head on a speaker-embedding cycle loss: clone, render, re-embed, cosine to the input audio embedding.
181 - [10k steps: dev cosine ~0.92; nearest stock voice baseline 0.857]
182 - Stage 2: the embedding alone missed texture and pitch; adjustments:
183 - Augmented pitch and speed. Added losses via a WavLM layer-statistics L1, a small log-F0 error, and spectral-tilt.
184 - [11k steps: dev cosine 0.934, DNSMOS 3.30]
185 - Stage 3: tuned the encoder side to match the blended predictors:
186 - Predictor half fixed to the blended rows, only the decoder half trained.
187 - [3k steps: dev cosine 0.932, DNSMOS 3.48]
188 - Stage 4: distilled the speaker encoder so enrollment needs no UniSpeech download:
189 - WeSpeaker ResNet34 plus a projection, trained on `1 - cos(student, teacher)` over LibriTTS-R crops with synthetic channel damage; the style head is untouched.
190 - [20k steps: hold-out dev-clean parity, DNSMOS equal; archival references sat about 0.05 behind the teacher]
191 - Stage 5: the student lagged the teacher on archival references; synthetic damage had plateaued:
192 - Mixed real public-domain archival broadcast audio into the distillation crops.
193 - [8k steps: archival cosine up 0.03-0.05, DNSMOS unchanged]
194
195 </details>
196
197 <details>
198 <summary>v0.2.0 training notes</summary>
199
200 - Stage 1: the blend under-predicted pitch range and sentence-final falls:
201 - Fitted a predictor-half delta per speaker (426; adds VoxPopuli-en and Emilia-YODAS EN)
202 - Targeted F0 mean, spread, IQR, pause fraction and speech time.
203 - Ridge head from reference F0 mean + spread to the fitted delta.
204 - [hold-out: F0 mean error 0.9 -> 0.45 st, spread 0.6 -> 0.35 st, IQR 0.9 -> 0.5 st]
205 - Stage 2: the student trailed the teacher on public-speaking and in-the-wild references:
206 - Added VoxPopuli and Emilia speakers to the distillation crops.
207 - [8k steps: hold-out cosine VoxPopuli 0.976 -> 0.986, Emilia 0.964 -> 0.983; WavLM cosine 0.941 -> 0.944, DNSMOS unchanged]
208 - Stage 3: the timbre half had only seen LibriTTS speakers:
209 - Repeated v0.1.0 stage 3 with VoxPopuli and Emilia speakers in the draw.
210 - [3k steps: WavLM cosine samples 0.927 -> 0.935, dev-clean 0.956 -> 0.959, VoxPopuli 0.926 -> 0.939, Emilia 0.931 -> 0.936; DNSMOS unchanged]
211
212 </details>
213
214 ### Zero-shot identity benchmark
215
216 Inno gets about a third of the normalized similarity range, twice as close as the nearest stock pack (with UTMOS intact and keeping Kokoro's stable natural style).
217
218 <figure>
219 <img
220 src="https://cdn-uploads.huggingface.co/production/uploads/6444c7a59174daa2f6934a16/FmO3P3W8k501WZgqUBuyM.webp"
221 alt="Zero-shot identity benchmark"
222 width="95%"
223 style="border: 1px solid black;"
224 />
225 <figcaption>
226 Benchmark Plot: Zero-shot speaker similarity comparison on LibriSpeech test-clean.
227 </figcaption>
228 </figure>
229
230 <details>
231 <summary><b>Full benchmark results and methodology</b></summary>
232
233 LibriSpeech test-clean, F5-TTS cross-sentence split (1127 utterances, 39 held-out speakers. Reference in, new sentence out, scored against the speaker's real recording.
234 - normalized: how far the render sits between sounding like a stranger (0) and a second real take of the speaker (1)
235 - RTF: RTX 4060 Ti
236
237 | system | SIM-o | normalized | ECAPA | UTMOS | RTF |
238 |---|---|---|---|---|---|
239 | ground truth (second recording) | 0.695 | 1.00 | 0.656 | 4.10 | |
240 | F5-TTS v1 base | 0.650 | 0.94 | 0.713 | 3.86 | 0.48 |
241 | StyleTTS2 (LibriTTS zero-shot) | 0.386 | 0.46 | 0.415 | 4.40 | 0.06 |
242 | **Inno v0.2** | 0.288 | 0.32 | 0.288 | 4.45 | 0.07 |
243 | OpenVoice v2 | 0.227 | 0.23 | 0.316 | 3.80 | 0.12 |
244 | Kokoro, nearest stock pack | 0.167 | 0.15 | 0.215 | 4.25 | 0.06 |
245
246 StyleTTS2 uses a related architecture with an unfrozen decoder, a possible ceiling for this style space.
247
248 - StyleTTS2: alpha 0.3, beta 0.7, 5 steps.
249 - OpenVoice v2: MeloTTS EN-US, tau 0.3.
250 - F5-TTS: 32 NFE, CFG 2.
251
252 </details>
253
254 <details>
255 <summary><b>v0.1 vs v0.2 shipped-weight comparison</b></summary>
256
257 Measured on the shipped weights through the stock `KPipeline`: one fixed passage rendered for the seven sample voices plus ten hold-out dev-clean speakers, F0 by Praat over 6 s chunks with the reference's tracking ceiling applied to both sides.
258
259 | metric | v0.1 blend only | v0.2 with head |
260 |---|---|---|
261 | speaker cosine (WavLM-sv), samples / dev-clean | 0.926 / 0.956 | 0.935 / 0.959 |
262 | F0 mean, render minus reference | 0.65 st mean abs | 0.8 st mean abs |
263 | F0 spread, render minus reference | 0.65 st mean abs | 0.5 st mean abs |
264 | DNSMOS | 3.50 | 3.50 |
265
266 The head's gain is spread; on F0 mean it scatters more per speaker and flips the signed bias from -0.2 to +0.3 st.
267
268 </details>
269
270 ## Licenses and credits
271
272 Adapter weights and code: Apache-2.0, except the baked speaker encoder (`enc.*` in `model.safetensors`), which is CC BY-SA 3.0 (see below)
273 - Built on [hexgrad/Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M) (Apache-2.0, StyleTTS2 architecture)
274 - The speaker encoder is a [WeSpeaker](https://github.com/wenet-e2e/wespeaker) ResNet34 (code Apache-2.0)
275 - initialised from the [VoxCeleb-trained ResNet34-LM weights](https://huggingface.co/Wespeaker/wespeaker-voxceleb-resnet34-LM) (CC BY 4.0)
276 - distilled from [microsoft/unispeech-sat-base-plus-sv](https://huggingface.co/microsoft/unispeech-sat-base-plus-sv)
277 - (Microsoft UniSpeech, [CC BY-SA 3.0 on GitHub](https://github.com/microsoft/UniSpeech/blob/main/LICENSE)); encoder weights under CC BY-SA 3.0
278 - Training data (no audio redistributed beyond the samples):
279 - [LibriTTS-R](https://www.openslr.org/141/) (CC BY 4.0), also the LibriTTS sample references
280 - [VoxPopuli](https://github.com/facebookresearch/voxpopuli) (CC0)
281 - [Emilia-YODAS](https://huggingface.co/datasets/amphion/Emilia-Dataset) EN (CC BY 4.0)
282 - Sample references:
283 - Amelia Earhart from ["Speech by Amelia Earhart"](https://www.loc.gov/item/afccal000004/), Library of Congress AFC 1986/022
284 - Jane Goodall from [BBC Woman's Hour excerpt](https://commons.wikimedia.org/wiki/File:Jane_Goodall_BBC_Radio4_Woman%27s_Hour_26_Jan_2010_p00tr6ks.flac), BBC, CC BY 3.0
285 - Vincent Price public domain from ["The House On Haunted Hill"](https://archive.org/details/The_House_On_Haunted_Hill)
286
287 Only clone voices you have permission to clone, even the shallow cloning offered via this model and technique.
288