README.md
| 1 | --- |
| 2 | base_model: hexgrad/Kokoro-82M |
| 3 | base_model_relation: adapter |
| 4 | pipeline_tag: text-to-speech |
| 5 | license: apache-2.0 |
| 6 | language: |
| 7 | - en |
| 8 | datasets: |
| 9 | - parler-tts/libritts_r_filtered |
| 10 | - facebook/voxpopuli |
| 11 | - amphion/Emilia-Dataset |
| 12 | tags: |
| 13 | - kokoro |
| 14 | - voice-cloning |
| 15 | - adapter |
| 16 | - styletts2 |
| 17 | - text-to-speech |
| 18 | --- |
| 19 | |
| 20 | # kokoro-inno-clone-tuner |
| 21 | |
| 22 |  |
| 23 | |
| 24 | ```python |
| 25 | pip install inno-kokoro |
| 26 | ``` |
| 27 | |
| 28 | Zero-shot voice tuner for [Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M). |
| 29 | |
| 30 | Outputs base Kokoro compatible voice packs @ `[510, 1, 256]`. ([write-up](https://remsky.art/writeups/kokoro-fastapi/inno-clone/)) |
| 31 | |
| 32 | |
| 33 | ## Samples |
| 34 | |
| 35 | Same passage for every voice, enrolled from the references. LibriTTS-R speakers are `dev-clean` held out from training. |
| 36 | |
| 37 | | voice | tuned | reference | |
| 38 | |---|---|---| |
| 39 | | am_libritts1272 | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/libritts_1272_clone.wav"></audio> | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/libritts_1272_ref.wav"></audio> | |
| 40 | | af_libritts5338 | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/libritts_5338_clone.wav"></audio> | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/libritts_5338_ref.wav"></audio> | |
| 41 | | am_libritts6241 | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/libritts_6241_clone.wav"></audio> | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/libritts_6241_ref.wav"></audio> | |
| 42 | | af_ameliaearhart | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/amelia_earhart_clone.wav"></audio> | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/amelia_earhart_ref.wav"></audio> | |
| 43 | | bf_janegoodall | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/jane_goodall_clone.wav"></audio> | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/jane_goodall_ref.wav"></audio> | |
| 44 | | am_vincentprice | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/vincent_price_clone.wav"></audio> | <audio controls src="https://huggingface.co/Remsky/kokoro-inno-clone-tuner/resolve/main/samples/vincent_price_ref.wav"></audio> | |
| 45 | |
| 46 | |
| 47 | |
| 48 | ## Usage |
| 49 | |
| 50 | Code available on Github: [remsky/inno-kokoro](https://github.com/remsky/inno-kokoro). |
| 51 | |
| 52 | Integrated into [Kokoro-FastAPI](https://github.com/remsky/Kokoro-FastAPI) (v0.9.0+) |
| 53 | |
| 54 | <details open> |
| 55 | <summary>PyTorch, with <a href="https://github.com/hexgrad/kokoro">kokoro</a></summary> |
| 56 | |
| 57 | ```bash |
| 58 | pip install "inno-kokoro[torch]" |
| 59 | ``` |
| 60 | |
| 61 | ```python |
| 62 | from inno_kokoro.enroll import Tuner, enroll, read |
| 63 | import torch |
| 64 | from kokoro import KPipeline |
| 65 | |
| 66 | tuner = Tuner(backend="torch") # caches model from HF on first use |
| 67 | pack, _ = enroll(*read("my_ref.wav"), tuner, return_pt=True) # [510, 1, 256] |
| 68 | torch.save(pack, "am_me.pt") |
| 69 | |
| 70 | pipe = KPipeline(lang_code="a") |
| 71 | wav = next(pipe("Hello from a tuned voice.", voice=pack)).audio |
| 72 | ``` |
| 73 | |
| 74 | </details> |
| 75 | |
| 76 | <details> |
| 77 | <summary>ONNX (no torch), with <a href="https://github.com/thewh1teagle/kokoro-onnx">kokoro-onnx</a></summary> |
| 78 | |
| 79 | ```bash |
| 80 | pip install inno-kokoro kokoro-onnx |
| 81 | ``` |
| 82 | |
| 83 | ```python |
| 84 | from inno_kokoro.enroll import Tuner, enroll, read |
| 85 | import numpy as np |
| 86 | from kokoro_onnx import Kokoro |
| 87 | |
| 88 | tuner = Tuner(backend="onnx") # downloads model.safetensors + model.onnx on first use |
| 89 | pack, _ = enroll(*read("my_ref.wav"), tuner) # [510, 1, 256] float32 NumPy |
| 90 | np.save("am_me.npy", pack) |
| 91 | |
| 92 | kokoro = Kokoro("kokoro-v1.0.onnx", "voices-v1.0.bin") # from the kokoro-onnx releases |
| 93 | wav, sr = kokoro.create("Hello from a tuned voice.", voice=pack, lang="en-us") |
| 94 | ``` |
| 95 | |
| 96 | </details> |
| 97 | |
| 98 | Notes: |
| 99 | - prefix names with stock accent+gender codes for use in other libraries: e.g `af_`, `am_`, `bf_`, `bm_` |
| 100 | - pitch ceiling is set automatically |
| 101 | - `enroll(..., fmax=180)`: can override when needed (archival sources, etc) |
| 102 | |
| 103 | ### ONNX |
| 104 | |
| 105 | `model.onnx`: the speaker encoder and style head for onnxruntime (fp32, 31.5 MB). Used by `Tuner(backend="onnx")` from inno-kokoro 0.4.0. Same packs as torch, about 2x faster on CPU. |
| 106 | |
| 107 | <details> |
| 108 | <summary><b>Graph details</b></summary> |
| 109 | |
| 110 | - input `fbank` [1, frames, 80] float32: kaldi log-mel of the first 30 s at 16 kHz (`inno_kokoro/fbank.py`) |
| 111 | - outputs `embedding` [1, 512] (unit, UniSpeech-SAT-sv space) and `style` [1, 256] |
| 112 | - opset 20, standard ops only, no DFT or custom ops |
| 113 | - matches torch within 5.4e-7 on 540 clips ([parity check](onnx_parity.md)) |
| 114 | - exported with `export_onnx.py` (inno-kokoro `tools/export_onnx.py` at b442727) |
| 115 | |
| 116 | | reference | onnxruntime | torch | |
| 117 | |---|---|---| |
| 118 | | 8 s | 100 ms | 165 ms | |
| 119 | | 30 s | 410 ms | 890 ms | |
| 120 | |
| 121 | CPU (best of 5 runs) |
| 122 | |
| 123 | </details> |
| 124 | |
| 125 | ### Architecture |
| 126 | |
| 127 | A 5 to 30 s reference clip in, a stock-shaped Kokoro voice pack out. Everything loads from `model.safetensors`, no extra downloads. |
| 128 | |
| 129 | <figure> |
| 130 | <img |
| 131 | src="https://cdn-uploads.huggingface.co/production/uploads/6444c7a59174daa2f6934a16/XKJQUMieuxKyu5xKu9dc3.png" |
| 132 | alt="Inno-Kokoro enrollment architecture" |
| 133 | width="95%" |
| 134 | /> |
| 135 | <figcaption>Enrollment Architecture Diagram: reference audio to Kokoro-compatible voice pack.</figcaption> |
| 136 | </figure> |
| 137 | |
| 138 | <details> |
| 139 | <summary><b>Architecture details</b></summary> |
| 140 | |
| 141 | Enrollment embeds an input audio sample via the baked speaker encoder and maps through `model.safetensors` to build the encoder side; prosody aspects are measured to build the predictor half. |
| 142 | |
| 143 | - **speaker encoder**: a 6.6M-param WeSpeaker ResNet34 distilled from [**UniSpeech-SAT-sv**](https://huggingface.co/microsoft/unispeech-sat-base-plus-sv) embeddings |
| 144 | - matches within 0.002 speaker cosine on hold-out dev-clean (380mb fp32 -> 14mb fp16) |
| 145 | - **style head**: the decoder (timbre) half of the pack comes from the speaker embedding (512) mapped to Kokoro style (256). Shifts along a learned spectral-tilt direction by the reference's measured tilt. |
| 146 | - **prosody blend**: the predictor (prosody) half uses the stock Kokoro packs as a foundation: |
| 147 | - Blended by nonnegative least squares to the reference F0 mean, F0 spread and syllable rate. |
| 148 | - Packs more than 4 st from the reference's pitch are excluded. |
| 149 | - Higher graded packs are weighted as preferred during selection. |
| 150 | - **prosody head** (v0.2): a linear adjust on the blend from the reference's F0 mean and spread, |
| 151 | - fitted to deltas optimized through the frozen predictor. `enroll(..., head=False)` skips it. |
| 152 | |
| 153 | Enrollment runs in about 0.05 s per second of reference on CPU (a 30 s reference in 1.4 s) and 0.1-0.3 s total on a GPU, after the one-time model load. |
| 154 | |
| 155 | Total model size @ fp16: ~24mb |
| 156 | |
| 157 | </details> |
| 158 | |
| 159 | ## Limits |
| 160 | |
| 161 | - English trained (LibriTTS-R) currently |
| 162 | - Some texture/identity is traded to maintain audio quality and stability within Kokoro's range. |
| 163 | - Rythym is a best-effort match to the speaker. |
| 164 | - Prosody is a convex blend of stock prosody by pitch, spread and pace. Outside of that range, it will only pin to the nearest edge |
| 165 | - Reference clip requirements: |
| 166 | - 3 second minimum, up to a max of 30s |
| 167 | - Reasonably free of audio artifacts |
| 168 | - Single speaker |
| 169 | |
| 170 | ## Training and Metrics |
| 171 | |
| 172 | All stages: Kokoro frozen; LibriTTS-R train-clean-100 unless noted. |
| 173 | |
| 174 | Trained via HF Jobs `a10g-small`; under $20 in GPU time: |
| 175 | |
| 176 | <details> |
| 177 | <summary>v0.1.0 training notes</summary> |
| 178 | |
| 179 | - Stage 1: mapped speaker embedding to timbre: |
| 180 | - Style head on a speaker-embedding cycle loss: clone, render, re-embed, cosine to the input audio embedding. |
| 181 | - [10k steps: dev cosine ~0.92; nearest stock voice baseline 0.857] |
| 182 | - Stage 2: the embedding alone missed texture and pitch; adjustments: |
| 183 | - Augmented pitch and speed. Added losses via a WavLM layer-statistics L1, a small log-F0 error, and spectral-tilt. |
| 184 | - [11k steps: dev cosine 0.934, DNSMOS 3.30] |
| 185 | - Stage 3: tuned the encoder side to match the blended predictors: |
| 186 | - Predictor half fixed to the blended rows, only the decoder half trained. |
| 187 | - [3k steps: dev cosine 0.932, DNSMOS 3.48] |
| 188 | - Stage 4: distilled the speaker encoder so enrollment needs no UniSpeech download: |
| 189 | - WeSpeaker ResNet34 plus a projection, trained on `1 - cos(student, teacher)` over LibriTTS-R crops with synthetic channel damage; the style head is untouched. |
| 190 | - [20k steps: hold-out dev-clean parity, DNSMOS equal; archival references sat about 0.05 behind the teacher] |
| 191 | - Stage 5: the student lagged the teacher on archival references; synthetic damage had plateaued: |
| 192 | - Mixed real public-domain archival broadcast audio into the distillation crops. |
| 193 | - [8k steps: archival cosine up 0.03-0.05, DNSMOS unchanged] |
| 194 | |
| 195 | </details> |
| 196 | |
| 197 | <details> |
| 198 | <summary>v0.2.0 training notes</summary> |
| 199 | |
| 200 | - Stage 1: the blend under-predicted pitch range and sentence-final falls: |
| 201 | - Fitted a predictor-half delta per speaker (426; adds VoxPopuli-en and Emilia-YODAS EN) |
| 202 | - Targeted F0 mean, spread, IQR, pause fraction and speech time. |
| 203 | - Ridge head from reference F0 mean + spread to the fitted delta. |
| 204 | - [hold-out: F0 mean error 0.9 -> 0.45 st, spread 0.6 -> 0.35 st, IQR 0.9 -> 0.5 st] |
| 205 | - Stage 2: the student trailed the teacher on public-speaking and in-the-wild references: |
| 206 | - Added VoxPopuli and Emilia speakers to the distillation crops. |
| 207 | - [8k steps: hold-out cosine VoxPopuli 0.976 -> 0.986, Emilia 0.964 -> 0.983; WavLM cosine 0.941 -> 0.944, DNSMOS unchanged] |
| 208 | - Stage 3: the timbre half had only seen LibriTTS speakers: |
| 209 | - Repeated v0.1.0 stage 3 with VoxPopuli and Emilia speakers in the draw. |
| 210 | - [3k steps: WavLM cosine samples 0.927 -> 0.935, dev-clean 0.956 -> 0.959, VoxPopuli 0.926 -> 0.939, Emilia 0.931 -> 0.936; DNSMOS unchanged] |
| 211 | |
| 212 | </details> |
| 213 | |
| 214 | ### Zero-shot identity benchmark |
| 215 | |
| 216 | Inno gets about a third of the normalized similarity range, twice as close as the nearest stock pack (with UTMOS intact and keeping Kokoro's stable natural style). |
| 217 | |
| 218 | <figure> |
| 219 | <img |
| 220 | src="https://cdn-uploads.huggingface.co/production/uploads/6444c7a59174daa2f6934a16/FmO3P3W8k501WZgqUBuyM.webp" |
| 221 | alt="Zero-shot identity benchmark" |
| 222 | width="95%" |
| 223 | style="border: 1px solid black;" |
| 224 | /> |
| 225 | <figcaption> |
| 226 | Benchmark Plot: Zero-shot speaker similarity comparison on LibriSpeech test-clean. |
| 227 | </figcaption> |
| 228 | </figure> |
| 229 | |
| 230 | <details> |
| 231 | <summary><b>Full benchmark results and methodology</b></summary> |
| 232 | |
| 233 | LibriSpeech test-clean, F5-TTS cross-sentence split (1127 utterances, 39 held-out speakers. Reference in, new sentence out, scored against the speaker's real recording. |
| 234 | - normalized: how far the render sits between sounding like a stranger (0) and a second real take of the speaker (1) |
| 235 | - RTF: RTX 4060 Ti |
| 236 | |
| 237 | | system | SIM-o | normalized | ECAPA | UTMOS | RTF | |
| 238 | |---|---|---|---|---|---| |
| 239 | | ground truth (second recording) | 0.695 | 1.00 | 0.656 | 4.10 | | |
| 240 | | F5-TTS v1 base | 0.650 | 0.94 | 0.713 | 3.86 | 0.48 | |
| 241 | | StyleTTS2 (LibriTTS zero-shot) | 0.386 | 0.46 | 0.415 | 4.40 | 0.06 | |
| 242 | | **Inno v0.2** | 0.288 | 0.32 | 0.288 | 4.45 | 0.07 | |
| 243 | | OpenVoice v2 | 0.227 | 0.23 | 0.316 | 3.80 | 0.12 | |
| 244 | | Kokoro, nearest stock pack | 0.167 | 0.15 | 0.215 | 4.25 | 0.06 | |
| 245 | |
| 246 | StyleTTS2 uses a related architecture with an unfrozen decoder, a possible ceiling for this style space. |
| 247 | |
| 248 | - StyleTTS2: alpha 0.3, beta 0.7, 5 steps. |
| 249 | - OpenVoice v2: MeloTTS EN-US, tau 0.3. |
| 250 | - F5-TTS: 32 NFE, CFG 2. |
| 251 | |
| 252 | </details> |
| 253 | |
| 254 | <details> |
| 255 | <summary><b>v0.1 vs v0.2 shipped-weight comparison</b></summary> |
| 256 | |
| 257 | Measured on the shipped weights through the stock `KPipeline`: one fixed passage rendered for the seven sample voices plus ten hold-out dev-clean speakers, F0 by Praat over 6 s chunks with the reference's tracking ceiling applied to both sides. |
| 258 | |
| 259 | | metric | v0.1 blend only | v0.2 with head | |
| 260 | |---|---|---| |
| 261 | | speaker cosine (WavLM-sv), samples / dev-clean | 0.926 / 0.956 | 0.935 / 0.959 | |
| 262 | | F0 mean, render minus reference | 0.65 st mean abs | 0.8 st mean abs | |
| 263 | | F0 spread, render minus reference | 0.65 st mean abs | 0.5 st mean abs | |
| 264 | | DNSMOS | 3.50 | 3.50 | |
| 265 | |
| 266 | The head's gain is spread; on F0 mean it scatters more per speaker and flips the signed bias from -0.2 to +0.3 st. |
| 267 | |
| 268 | </details> |
| 269 | |
| 270 | ## Licenses and credits |
| 271 | |
| 272 | Adapter weights and code: Apache-2.0, except the baked speaker encoder (`enc.*` in `model.safetensors`), which is CC BY-SA 3.0 (see below) |
| 273 | - Built on [hexgrad/Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M) (Apache-2.0, StyleTTS2 architecture) |
| 274 | - The speaker encoder is a [WeSpeaker](https://github.com/wenet-e2e/wespeaker) ResNet34 (code Apache-2.0) |
| 275 | - initialised from the [VoxCeleb-trained ResNet34-LM weights](https://huggingface.co/Wespeaker/wespeaker-voxceleb-resnet34-LM) (CC BY 4.0) |
| 276 | - distilled from [microsoft/unispeech-sat-base-plus-sv](https://huggingface.co/microsoft/unispeech-sat-base-plus-sv) |
| 277 | - (Microsoft UniSpeech, [CC BY-SA 3.0 on GitHub](https://github.com/microsoft/UniSpeech/blob/main/LICENSE)); encoder weights under CC BY-SA 3.0 |
| 278 | - Training data (no audio redistributed beyond the samples): |
| 279 | - [LibriTTS-R](https://www.openslr.org/141/) (CC BY 4.0), also the LibriTTS sample references |
| 280 | - [VoxPopuli](https://github.com/facebookresearch/voxpopuli) (CC0) |
| 281 | - [Emilia-YODAS](https://huggingface.co/datasets/amphion/Emilia-Dataset) EN (CC BY 4.0) |
| 282 | - Sample references: |
| 283 | - Amelia Earhart from ["Speech by Amelia Earhart"](https://www.loc.gov/item/afccal000004/), Library of Congress AFC 1986/022 |
| 284 | - Jane Goodall from [BBC Woman's Hour excerpt](https://commons.wikimedia.org/wiki/File:Jane_Goodall_BBC_Radio4_Woman%27s_Hour_26_Jan_2010_p00tr6ks.flac), BBC, CC BY 3.0 |
| 285 | - Vincent Price public domain from ["The House On Haunted Hill"](https://archive.org/details/The_House_On_Haunted_Hill) |
| 286 | |
| 287 | Only clone voices you have permission to clone, even the shallow cloning offered via this model and technique. |
| 288 | |