README.md · VieNeu-TTS-v3-Turbo

1

---

2

license: apache-2.0

3

datasets:

4

- pnnbao-ump/VieNeu-TTS-10k-ENVI

5

language:

6

- vi

7

- en

8

pipeline_tag: text-to-speech

9

tags:

10

- voice-cloning

11

- code-switching

12

- podcast

13

- emotion-control

14

- 48khz

15

---

16

17

# 🦜 VieNeu-TTS v3 Turbo

18

19

[![GitHub](https://img.shields.io/badge/GitHub-Repository-blue)](https://github.com/pnnbao97/VieNeu-TTS)

20

[![Model](https://img.shields.io/badge/Hugging%20Face-Model-yellow)](https://huggingface.co/pnnbao-ump/VieNeu-TTS-v3-Turbo)

21

[![PyPI](https://img.shields.io/badge/PyPI-vieneu%203.7.1-blue?logo=pypi&logoColor=white)](https://pypi.org/project/vieneu/)

22

[![Discord](https://img.shields.io/badge/Discord-Join%20Us-5865F2?logo=discord&logoColor=white)](https://discord.gg/yJt8kzjzWZ)

23

24

## Overview

25

26

<video controls src="https://cdn-uploads.huggingface.co/production/uploads/68b923a86c86c127a1975eda/paPqSDpwFGrKtIZrqaEg4.mp4" width="100%"></video>

27

28

**VieNeu-TTS v3 Turbo** is the next generation of Vietnamese TTS — **48 kHz** high-fidelity speech, **23 built-in preset voices** across three regions (North / Central / South), **instant voice cloning**, **real-time streaming** with an **OpenAI-compatible API** (16 concurrent streams on one RTX 3060), inline **emotion cues**, and seamless **bilingual (En–Vi) code-switching**.

29

30

The reference implementation is the **`vieneu` Python SDK (v3.7.1)**. Its minimal install is **torch-free**: on CPU everything runs on **ONNX Runtime** (PyTorch is never imported), and on a CUDA machine it auto-switches to the PyTorch engine with **automatic batching** and a **continuous-batching stream scheduler** — same API, no code change.

31

32

> [!IMPORTANT]

33

> **What's new in SDK v3.7:**

34

> - **GPU streaming with continuous batching** — `infer_stream` on CUDA runs on one CUDA graph shared by every caller: first audio in **~115 ms**, **16 concurrent streams under 200 ms** on a single RTX 3060 (32 max).

35

> - **OpenAI-compatible streaming API + Docker** — `POST /v1/audio/speech` (`pcm`/`wav`, chunked or SSE) works with the OpenAI SDK, Pipecat, LiveKit; `docker compose --profile api-gpu` / `api-cpu`. Numbers, sizing and CPU-only notes: [docs/streaming.md](https://github.com/pnnbao97/VieNeu-TTS/blob/main/docs/streaming.md).

36

> - **23 preset voices** covering North / Central / South, both genders and several reading characters (default: `Minh Quân`).

37

> - **LoRA fine-tuning** of v3 Turbo on one consumer GPU.

38

> - Since 3.3: torch-free voice cloning on CPU, optional **int8** CPU backbone (`Vieneu(precision="int8")`, ~2× faster; fp32 is the default), sliding-window repetition penalty.

39

40

## 🏗️ Architecture & Credits

41

42

The **VieNeu-TTS v3 Turbo architecture is an original design by the author, Phạm Nguyễn Ngọc Bảo**, and is **trained from scratch** on ~10,000 hours of English–Vietnamese speech — it is **not** a fine-tune, distillation, or adaptation of any existing TTS model.

43

44

- **Model architecture & training:** designed and trained from scratch by **Phạm Nguyễn Ngọc Bảo** — https://github.com/pnnbao97

45

- **Audio codec:** [MOSS-Audio-Tokenizer-Nano](https://huggingface.co/OpenMOSS-Team/MOSS-Audio-Tokenizer-Nano) (OpenMOSS-Team) — 48 kHz neural audio codec.

46

- **Phonemizer:** [sea-g2p](https://github.com/pnnbao97/sea-g2p) — fast Vietnamese/English grapheme-to-phoneme, also by the author.

47

48

Tác giả: **Phạm Nguyễn Ngọc Bảo**

49

50

---

51

52

## 🔥 Quick Start (Web UI)

53

54

```bash

55

git clone https://github.com/pnnbao97/VieNeu-TTS.git

56

cd VieNeu-TTS

57

```

58

59

- **Option 1: CPU & macOS (minimal, torch-free) — recommended** — runs **v3 Turbo via ONNX**

60

61

```bash

62

uv sync

63

```

64

65

  > ⚡ Use `uv sync`, not `pip install`, for the fastest CPU inference — it reproduces the locked environment with the optimized ONNX Runtime build. On Apple Silicon this ONNX/CPU path is **faster** than the MPS/PyTorch build.

66

67

- **Option 2: GPU (CUDA ≥ 12.8)** — **v3 Turbo on GPU (PyTorch)**: batched generation + multi-stream real-time streaming

68

69

```bash

70

uv sync --extra cuda

71

```

72

73

**Start the Web UI:**

74

75

```bash

76

uv run vieneu-web

77

```

78

79

The UI opens at `http://127.0.0.1:7860` with a **Default voice** tab, a **Voice Cloning** tab, and a **Conversation** tab (batched multi-speaker podcasts).

80

81

---

82

83

## 📦 Using the Python SDK (`vieneu`)

84

85

**CPU (default)** — torch-free, runs v3 Turbo via ONNX Runtime. Most users want this:

86

87

```bash

88

pip install vieneu

89

```

90

91

**GPU (CUDA)** — only if you have an NVIDIA GPU; install a CUDA build of PyTorch yourself first:

92

93

```bash

94

pip install torch==2.8.0 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu128

95

pip install "transformers==4.57.6" # Qwen3 backbone + MOSS codec (pinned — most stable)

96

pip install vieneu

97

```

98

99

> ℹ️ **When is GPU actually worth it?** Two cases: **bulk / long-form synthesis** (many chunks share one batched forward) and **serving many listeners** (the stream scheduler batches every `infer_stream` call — 16 concurrent real-time streams on an RTX 3060). For a **single short, interactive call** the torch-free **CPU/ONNX** path is fast enough (first audio ~140 ms with int8, ~300 ms with fp32) and needs no GPU.

100

101

### Full features guide

102

103

```python

104

from vieneu import Vieneu

105

from time import time

106

107

# Default = v3 Turbo (48 kHz). CPU → ONNX (torch-free, fp32); GPU → PyTorch (auto-detected).

108

tts = Vieneu() # fp32 backbone (default, max fidelity)

109

# tts = Vieneu(precision="int8") # CPU only: ~2x faster, needs a CPU with VNNI

110

111

text = """[cười] Trời ơi, cái giọng nó tự nhiên mà nó mượt mà dã man, nghe không khác gì người thật luôn. Giờ thì tha hồ mà quẩy content với cả kho giọng nói đa dạng, đủ mọi sắc thái biểu cảm. Mọi người bật loa lên rồi cùng trải nghiệm thử với mình nhé!"""

112

113

# 1. Default voice (Minh Quân) — 48 kHz, no reference needed

114

start = time()

115

audio = tts.infer(text)

116

tts.save(audio, "output.wav")

117

print(f"Time taken: {time() - start:.2f} seconds")

118

119

# 2. Built-in voices by name

120

for label, voice_id in tts.list_preset_voices():

121

print(label, voice_id)

122

audio = tts.infer("Mình là Xuân Vĩnh nè!", voice="Xuân Vĩnh")

123

tts.save(audio, "output_xuan_vinh.wav")

124

125

# 3. Emotion / non-verbal cues — EXPERIMENTAL: [cười] [thở dài] [hắng giọng]

126

audio = tts.infer("Nghe hay quá đi [cười]. Để mình nói tiếp [hắng giọng].", voice="Phạm Tuyên")

127

128

# 4. Instant voice cloning from a 3–8s reference clip (works on the torch-free CPU install too)

129

audio = tts.infer("Đây là giọng được nhân bản tức thì.", ref_audio="my_voice.wav", denoise=True)

130

```

131

132

> [!TIP]

133

> A **temperature around 0.8** gives the most stable result for v3 Turbo. Higher values add expressiveness but can be less stable.

134

135

### 🔊 Real-time streaming

136

137

v3 Turbo streams frame by frame on both backends. **GPU**: first audio in **~115 ms**, and every `infer_stream` call shares one CUDA graph (continuous batching) — **16 concurrent streams stay real-time** on a single RTX 3060 (RTF ≈ 0.5–0.6, 32 streams max). **CPU** (ONNX): first audio ~140 ms (int8) / ~300 ms (fp32), RTF < 1, one stream (two with int8).

138

139

```python

140

vieneu = Vieneu() # GPU → PyTorch + stream scheduler; no GPU → ONNX/CPU

141

for chunk in vieneu.infer_stream("Xin chào! Đây là chế độ streaming của VieNeu, phát tới đâu nghe tới đó.", voice="Mai Anh"):

142

play(chunk) # np.float32 @ 48 kHz, play/write as it arrives

143

```

144

145

**OpenAI-compatible API** — `POST /v1/audio/speech` (`pcm`/`wav`, chunked body or SSE), drop-in for the OpenAI SDK, Pipecat, LiveKit:

146

147

```bash

148

uv run python -m apps.openai_speech # → http://localhost:8000

149

# (or: docker compose -f docker/docker-compose.yml --profile api-gpu up)

150

```

151

152

```python

153

from openai import OpenAI

154

client = OpenAI(base_url="http://localhost:8000/v1", api_key="x")

155

with client.audio.speech.with_streaming_response.create(

156

model="vieneu-v3-turbo", voice="Mai Anh", response_format="pcm",

157

input="Xin chào! Đây là chế độ streaming của VieNeu, phát tới đâu nghe tới đó.",

158

) as r:

159

for chunk in r.iter_bytes(4096): # s16le 48 kHz mono, as it is generated

160

play(chunk)

161

```

162

163

All measurements (TTFA / RTF / streams vs `max_streams` on an RTX 3060), estimates for smaller GPUs and the CPU-only numbers: [docs/streaming.md](https://github.com/pnnbao97/VieNeu-TTS/blob/main/docs/streaming.md).

164

165

### ⚡ Batched generation (GPU)

166

167

`infer_batch()` runs many texts in **one batched forward** — same API on every backend (on CPU it still works, just sequentially). The batch caps at `max_batch_size` (default 32); pass `batch_size=1` to disable. A single long `infer()` also auto-batches its own chunks.

168

169

```python

170

audios = vieneu.infer_batch(texts, voice="Mai Anh") # or infer_batch(..., batch_size=64)

171

```

172

173

### 🦜 Voice cloning & saved voices

174

175

```python

176

# Clone from a 3–8s clip; the reference is auto-denoised and trimmed to ≤ 8s

177

audio = vieneu.infer("Chào bạn, đây là giọng của tôi.", ref_audio="voice.wav", denoise=True)

178

179

# Enroll once, then reuse by name like a built-in voice

180

vieneu.add_voice("Giọng của tôi", "voice.wav")

181

audio = vieneu.infer("Câu này dùng giọng đã lưu.", voice="Giọng của tôi")

182

183

# Just clean up a clip (no synthesis)

184

wav, sr = vieneu.denoise("noisy.wav", out_path="clean.wav")

185

```

186

187

> `denoise`, `add_voice` and cloning work on **every** backend, including the torch-free CPU/ONNX install.

188

189

### ⚠️ Reading style is deprecated

190

191

`style` is **still accepted** by `infer`, `infer_stream`, `infer_batch` and `add_voice` so existing code keeps running, but it is **ignored** on v3 Turbo: the reading style is already baked into the reference itself (the speaker embedding + reference codes of the preset voice or of your cloned clip). Pick the reading character through the **voice** instead.

192

193

---

194

195

## 🎭 Preset Voices (23)

196

197

Call any of them by name via `voice="<name>"` — no reference audio required. Default voice: **Minh Quân**. (Generated from the SDK's `voices_v3_turbo.json`; `vieneu.list_preset_voices()` is authoritative.)

198

199

200

|---|---|---|---|---|---|---|---|---|

201

| Anh Khôi | Bắc | M | Storytelling | | Đoan Trang | Bắc | F | Natural |

202

| Mai Anh | Bắc | F | News | | Ngọc Trân | Trung | F | Natural |

203

| **Minh Quân** (default) | Bắc | M | Natural | | Quang Sơn | Trung | M | Natural |

204

| Minh Đức | Bắc | M | News | | Adam | Nam | M | Natural |

205

| Mạnh Dũng | Bắc | M | Natural | | Kim Thanh | Nam | F | Storytelling |

206

| Ngọc Huyền | Bắc | F | Natural | | Minh Triết | Nam | M | News |

207

| Ngọc Linh | Bắc | F | Storytelling | | Mỹ Duyên | Nam | F | Storytelling |

208

| Phạm Tuyên | Bắc | M | Natural | | Thái Sơn | Nam | M | Storytelling |

209

| Quỳnh Anh | Bắc | F | Storytelling | | Thùy Dung | Nam | F | News |

210

| Thanh Bình | Bắc | M | Storytelling | | Thục Đoan | Nam | F | Storytelling |

211

| Trúc Ly | Bắc | F | Natural | | Đức Trí | Nam | M | Storytelling |

212

| Xuân Vĩnh | Bắc | M | Natural | | | | | |

213

214

For any other voice, use **voice cloning** with a short reference clip (`ref_audio="..."`).

215

216

---

217

218

## 🔬 Model Variants

219

220

221

| --- | --- | --- | --- | --- | --- |

222

| **VieNeu-TTS-v3-Turbo** *(default)* | ONNX (CPU) / PyTorch (GPU) | CPU/GPU | 48 kHz | ⭐⭐⭐⭐⭐ | **23 preset voices, cloning, streaming (GPU: 16 concurrent streams, OpenAI-compatible API), emotion cues, conversation, LoRA fine-tuning** |

223

| VieNeu-TTS-v2 | PyTorch | GPU/CPU | 24 kHz | ⭐⭐⭐⭐⭐ | Podcast, En-Vi code-switching |

224

| VieNeu-TTS-v2 (GGUF) | GGUF Q4 | CPU | 24 kHz | ⭐⭐⭐⭐ | Fastest on CPU, Podcast |

225

| VieNeu-TTS-v1 | PyTorch | GPU | 24 kHz | ⭐⭐⭐⭐ | Stable (Vi only) |

226

227

---

228

229

## 📜 Usage Rights & Licensing FAQ

230

231

**Does Apache-2.0 cover every artifact in this repository?**

232

Yes. The license applies to **all** artifacts shipped here — `model.safetensors`, the ONNX exports, configs and tokenizers, and the bundled **preset-voice assets** (speaker embeddings + reference codes in `voices_v3_turbo.json`).

233

234

**May I use the preset voices and the generated audio commercially?**

235

Yes. The bundled preset voices are distributed under the same Apache-2.0 license as the rest of the repository, and audio generated with them **may be used in commercial and monetized content** (voice-over, videos, products, services) — no additional license or fee.

236

237

**Did the speakers behind the preset voices consent to AI training and synthetic speech?**

238

Yes. The speakers (or rightsholders) behind the shipped preset-voice assets granted appropriate rights and consent for their voice data to be used in **AI training and synthetic speech generation**, which is what allows those assets to be distributed under Apache-2.0 for both non-commercial and commercial synthetic audio generation.

239

240

**What about the training dataset?**

241

The detailed internal data-collection and processing pipeline for the training corpus is **not publicly disclosed**, and the [VieNeu-TTS-10k-ENVI](https://huggingface.co/datasets/pnnbao-ump/VieNeu-TTS-10k-ENVI) dataset is gated. The confirmations above cover the **preset voices shipped in this repository** and the model weights released here, which are the artifacts you actually redistribute or generate audio with.

242

243

**Which preset list is authoritative?**

244

`vieneu.list_preset_voices()` at the version you have installed. This card documents **SDK v3.7.1 (23 voices, default `Minh Quân`)**; earlier revisions shipped fewer voices under partly different names (3.3.0: 20 voices, default `Adam`), so pin the SDK version if the exact roster matters to you.

245

246

**Third-party components** — all permissively licensed, keep their notices when redistributing:

247

- [MOSS-Audio-Tokenizer-Nano](https://huggingface.co/OpenMOSS-Team/MOSS-Audio-Tokenizer-Nano) (OpenMOSS-Team) — Apache-2.0.

248

- [sea-g2p](https://github.com/pnnbao97/sea-g2p) — phonemizer, by the same author as this project.

249

250

> [!WARNING]

251

> **Voice cloning is your responsibility.** The consent confirmation above covers the **bundled preset voices only**. If you clone a voice from your own reference clip, you must have the right to use that person's voice. Do not clone real people without their permission, and do not use this model to impersonate, defraud, or produce misleading content.

252

253

---

254

255

## License

256

257

This model package is distributed under [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0), matching the upstream model repository.

258

259

When you reuse, redistribute, or convert these assets, please keep the license notice and attribution intact for both:

260

261

- the original project: [pnnbao97/VieNeu-TTS](https://github.com/pnnbao97/VieNeu-TTS)

262

- this Hugging Face package: [pnnbao-ump/VieNeu-TTS-v3-Turbo](https://huggingface.co/pnnbao-ump/VieNeu-TTS-v3-Turbo)

263

264

If you bundle additional third-party assets, their own licenses still apply as well.

265

266

---

267

268

## 📑 Citation

269

270

```bibtex

271

@misc{vieneutts2026,

272

title = {VieNeu-TTS v3 Turbo: 48kHz Vietnamese Text-to-Speech with Instant Voice Cloning and Emotion Control},

273

author = {Pham Nguyen Ngoc Bao},

274

year = {2026},

275

publisher = {Hugging Face},

276

howpublished = {\url{https://huggingface.co/pnnbao-ump/VieNeu-TTS-v3-Turbo}}

277

}

278

```

279

280

---

281

282

**Made with ❤️ for the Vietnamese TTS community**

283