README.md · canary-180m-flash-gguf

1

---

2

license: cc-by-4.0

3

base_model: nvidia/canary-180m-flash

4

base_model_relation: quantized

5

library_name: transcribe.cpp

6

pipeline_tag: automatic-speech-recognition

7

language:

8

- en

9

- de

10

- es

11

- fr

12

tags:

13

- gguf

14

- transcribe.cpp

15

- asr

16

- speech-to-text

17

- canary

18

- multitask-aed

19

- encoder-decoder

20

- translation

21

transcribe_cpp:

22

schema_version: 2

23

wer_fleurs_de:

24

q8_0: 7.33

25

wer_fleurs_en:

26

q8_0: 5.98

27

wer_fleurs_es:

28

q8_0: 6.54

29

wer_fleurs_fr:

30

q8_0: 8.53

31

wer_librispeech_test_clean:

32

f32: 1.94

33

f16: 1.94

34

q8_0: 1.93

35

q6_k: 1.93

36

q5_k_m: 1.9

37

q4_k_m: 1.93

38

rtf_m4_max:

39

cpu: 78.91

40

metal: 153.46

41

rtf_ryzen_4750u:

42

cpu: 21.73

43

vulkan: 33.09

44

streaming: false

45

translate: true

46

lang_detect: false

47

timestamps: none

48

---

49

50

# canary-180m-flash: transcribe.cpp GGUF

51

52

GGUF conversions of [nvidia/canary-180m-flash](https://huggingface.co/nvidia/canary-180m-flash) for use

53

with [transcribe.cpp](https://github.com/handy-computer/transcribe.cpp).

54

55

Ported from upstream commit

56

[b12ab41](https://huggingface.co/nvidia/canary-180m-flash/commit/b12ab41),

57

pinned 2026-05-08.

58

Validated against the NeMo reference at transcribe.cpp commit

59

[db53eda](https://github.com/handy-computer/transcribe.cpp/tree/db53eda)

60

on 2026-05-08.

61

62

Offline multilingual speech-to-text and translation. A 182M-parameter

63

multitask AED with a 17-layer FastConformer encoder and a 4-layer

64

Transformer decoder. Supports automatic speech recognition in English,

65

German, Spanish, and French, and bidirectional EN↔{DE, ES, FR}

66

translation. Takes a

67

16 kHz mono WAV and produces a transcript. Not a streaming model;

68

word/segment timestamps are upstream-experimental and not exposed in

69

the v1 port.

70

71

72

## Downloads

73

74

75

| --- | --- | ---: | ---: |

76

| F32 | [canary-180m-flash-F32.gguf](https://huggingface.co/handy-computer/canary-180m-flash-gguf/resolve/main/canary-180m-flash-F32.gguf) | 756 MB | 1.94% |

77

| F16 | [canary-180m-flash-F16.gguf](https://huggingface.co/handy-computer/canary-180m-flash-gguf/resolve/main/canary-180m-flash-F16.gguf) | 382 MB | 1.94% |

78

| Q8_0 | [canary-180m-flash-Q8_0.gguf](https://huggingface.co/handy-computer/canary-180m-flash-gguf/resolve/main/canary-180m-flash-Q8_0.gguf) | 218 MB | 1.93% |

79

| Q6_K | [canary-180m-flash-Q6_K.gguf](https://huggingface.co/handy-computer/canary-180m-flash-gguf/resolve/main/canary-180m-flash-Q6_K.gguf) | 176 MB | 1.93% |

80

| Q5_K_M | [canary-180m-flash-Q5_K_M.gguf](https://huggingface.co/handy-computer/canary-180m-flash-gguf/resolve/main/canary-180m-flash-Q5_K_M.gguf) | 159 MB | 1.90% |

81

| Q4_K_M | [canary-180m-flash-Q4_K_M.gguf](https://huggingface.co/handy-computer/canary-180m-flash-gguf/resolve/main/canary-180m-flash-Q4_K_M.gguf) | 139 MB | 1.93% |

82

83

WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded.

84

85

Greedy decoding, no external LM. F32 reference baseline: 1.94%. On the same wavs,

86

NeMo's reference run produces 1.93% (one substitution difference out of ~27k

87

reference words), so the F32 port matches the reference framework at the noise

88

floor. NVIDIA's self-reported number on the upstream model card is 1.87%.

89

90

91

## Usage

92

93

Build transcribe.cpp from source:

94

95

```bash

96

git clone git@github.com:handy-computer/transcribe.cpp.git

97

cd transcribe.cpp

98

cmake -B build && cmake --build build

99

```

100

101

Run on a 16 kHz mono WAV:

102

103

```bash

104

build/bin/transcribe-cli \

105

-m canary-180m-flash-Q8_0.gguf \

106

input.wav

107

```

108

109

If your audio isn't already 16 kHz mono WAV, convert it first:

110

111

```bash

112

ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav

113

```

114

115

See the [transcribe.cpp model page](https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/canary-180m-flash.md) for performance

116

numbers, numerical validation, and reproduction steps.

117

118

## License

119

120

Inherited from the base model: **CC-BY-4.0**. See the

121

[upstream model card](https://huggingface.co/nvidia/canary-180m-flash) for full terms.

122

123

---

124

125

## Original Model Card

126

127

> The section below is reproduced from

128

> [nvidia/canary-180m-flash](https://huggingface.co/nvidia/canary-180m-flash) at commit

129

> `b12ab41` for offline reference. The upstream card is the

130

> authoritative source.

131

132

# Canary 180M Flash

133

134

<style>

135

img {

136

display: inline;

137

}

138

</style>

139

140

## Description:

141

NVIDIA NeMo Canary Flash [1] is a family of multilingual multi-tasking models based on Canary architecture [2] that achieves state-of-the art performance on multiple speech benchmarks. With 182 million parameters and an inference speed of more than 1200 RTFx (on open-asr-leaderboard sets), canary-180m-flash supports automatic speech-to-text recognition (ASR) in 4 languages (English, German, French, Spanish) and translation from English to German/French/Spanish and from German/French/Spanish to English with or without punctuation and capitalization (PnC).

142

Additionally, canary-180m-flash offers an experimental feature for word-level and segment-level timestamps in English, German, French, and Spanish.

143

This model is released under the permissive CC-BY-4.0 license and is available for commercial use.

144

145

146

## Model Architecture:

147

Canary is an encoder-decoder model with FastConformer [3] Encoder and Transformer Decoder [4]. With audio features extracted from the encoder, task tokens such as \<target language\>, \<task\>, \<toggle timestamps\> and \<toggle PnC\> are fed into the Transformer Decoder to trigger the text generation process. Canary uses a concatenated tokenizer [5] from individual SentencePiece [6] tokenizers of each language, which makes it easy to scale up to more languages. The canary-180m-flash model has 17 encoder layers and 4 decoder layers, leading to a total of 182M parameters. For more details about the architecture, please refer to [1].

148

149

## NVIDIA NeMo

150

151

To train, fine-tune or transcribe with canary-180m-flash, you will need to install [NVIDIA NeMo](https://github.com/NVIDIA/NeMo).

152

153

## How to Use this Model

154

155

The model is available for use in the NeMo framework [7], and can be used as a pre-trained checkpoint for inference or for fine-tuning on another dataset.

156

157

Please refer to [our tutorial](https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/Canary_Multitask_Speech_Model.ipynb) for more details.

158

159

A few inference examples listed below:

160

161

### Loading the Model

162

163

```python

164

from nemo.collections.asr.models import EncDecMultiTaskModel

165

# load model

166

canary_model = EncDecMultiTaskModel.from_pretrained('nvidia/canary-180m-flash')

167

# update decode params

168

decode_cfg = canary_model.cfg.decoding

169

decode_cfg.beam.beam_size = 1

170

canary_model.change_decoding_strategy(decode_cfg)

171

```

172

173

## Input:

174

**Input Type(s):** Audio 

175

**Input Format(s):** .wav or .flac files 

176

**Input Parameters(s):** 1D 

177

**Other Properties Related to Input:** 16000 Hz Mono-channel Audio, Pre-Processing Not Needed 

178

179

Input to canary-180m-flash can be either a list of paths to audio files or a jsonl manifest file.

180

181

### Inference with canary-180m-flash:

182

If the input is a list of paths, canary-180m-flash assumes that the audio is English and transcribes it. I.e., canary-180m-flash default behavior is English ASR.

183

```python

184

output = canary_model.transcribe(

185

['path1.wav', 'path2.wav'],

186

batch_size=16, # batch size to run the inference with

187

pnc='True', # generate output with Punctuation and Capitalization

188

)

189

190

predicted_text = output[0].text

191

192

```

193

194

canary-180m-flash can also predict word-level and segment-level timestamps

195

```python

196

output = canary_model.transcribe(

197

['filepath.wav'],

198

timestamps=True, # generate output with timestamps

199

)

200

201

predicted_text = output[0].text

202

word_level_timestamps = output[0].timestamp['word']

203

segment_level_timestamps = output[0].timestamp['segment']

204

205

```

206

To predict timestamps for audio files longer than 10 seconds, we recommend using the longform inference script (explained in the next section) with `chunk_len_in_secs=10.0`.

207

208

To use canary-180m-flash for transcribing other supported languages or perform Speech-to-Text translation or provide word-level timestamps, specify the input as jsonl manifest file, where each line in the file is a dictionary containing the following fields:

209

210

```yaml

211

# Example of a line in input_manifest.json

212

{

213

"audio_filepath": "/path/to/audio.wav", # path to the audio file

214

    "source_lang": "en",  # language of the audio input, set `source_lang`==`target_lang` for ASR, choices=['en','de','es','fr']

215

"target_lang": "en", # language of the text output, choices=['en','de','es','fr']

216

"pnc": "yes", # whether to have PnC output, choices=['yes', 'no']

217

"timestamp": "yes", # whether to output word-level timestamps, choices=['yes', 'no']

218

}

219

```

220

221

and then use:

222

```python

223

output = canary_model.transcribe(

224

"<path to input manifest file>",

225

batch_size=16, # batch size to run the inference with

226

)

227

```

228

229

### Longform inference with canary-180m-flash:

230

Canary models are designed to handle input audio smaller than 40 seconds. In order to handle longer audios, NeMo includes [speech_to_text_aed_chunked_infer.py](https://github.com/NVIDIA/NeMo/blob/main/examples/asr/asr_chunked_inference/aed/speech_to_text_aed_chunked_infer.py) script that handles chunking, performs inference on the chunked files, and stitches the transcripts.

231

232

The script will perform inference on all `.wav` files in `audio_dir`. Alternatively you can also pass a path to a manifest file as shown above. The decoded output will be saved at `output_json_path`.

233

234

```

235

python scripts/speech_to_text_aed_chunked_infer.py \

236

pretrained_name="nvidia/canary-180m-flash" \

237

audio_dir=$audio_dir \

238

output_filename=$output_json_path \

239

chunk_len_in_secs=40.0 \

240

batch_size=1 \

241

decoding.beam.beam_size=1 \

242

timestamps=False

243

```

244

245

**Note** that for longform inference with timestamps, it is recommended to use `chunk_len_in_secs` of 10 seconds.

246

247

248

## Output:

249

**Output Type(s):** Text 

250

**Output Format:** Text output as a string (w/ timestamps) depending on the task chosen for decoding 

251

**Output Parameters:** 1-Dimensional text string 

252

**Other Properties Related to Output:** May Need Inverse Text Normalization; Does Not Handle Special Characters 

253

254

255

## Software Integration:

256

**Runtime Engine(s):**

257

* NeMo - main 

258

259

**Supported Hardware Microarchitecture Compatibility:** 

260

* [NVIDIA Ampere] 

261

* [NVIDIA Blackwell] 

262

* [NVIDIA Jetson] 

263

* [NVIDIA Hopper] 

264

* [NVIDIA Lovelace] 

265

* [NVIDIA Pascal] 

266

* [NVIDIA Turing] 

267

* [NVIDIA Volta] 

268

269

**[Preferred/Supported] Operating System(s):** 

270

* [Linux] 

271

* [Linux 4 Tegra] 

272

* [Windows] 

273

274

## Model Version(s):

275

canary-180m-flash 

276

277

278

# Training and Evaluation Datasets:

279

280

## Training Dataset:

281

282

The canary-180m-flash model is trained on a total of 85K hrs of speech data. It consists of 31K hrs of public data, 20K hrs collected by [Suno](https://suno.ai/), and 34K hrs of in-house data.

283

The datasets below include conversations, videos from the web, and audiobook recordings.

284

285

**Data Collection Method:**

286

* Human 

287

288

**Labeling Method:**

289

* Hybrid: Human, Automated 

290

291

The constituents of public data are as follows.

292

293

#### English (25.5k hours)

294

- Librispeech 960 hours

295

- Fisher Corpus

296

- Switchboard-1 Dataset

297

- WSJ-0 and WSJ-1

298

- National Speech Corpus (Part 1, Part 6)

299

- VCTK

300

- VoxPopuli (EN)

301

- Europarl-ASR (EN)

302

- Multilingual Librispeech (MLS EN) - 2,000 hour subset

303

- Mozilla Common Voice (v7.0)

304

- People's Speech - 12,000 hour subset

305

- Mozilla Common Voice (v11.0) - 1,474 hour subset

306

307

#### German (2.5k hours)

308

- Mozilla Common Voice (v12.0) - 800 hour subset

309

- Multilingual Librispeech (MLS DE) - 1,500 hour subset

310

- VoxPopuli (DE) - 200 hr subset

311

312

#### Spanish (1.4k hours)

313

- Mozilla Common Voice (v12.0) - 395 hour subset

314

- Multilingual Librispeech (MLS ES) - 780 hour subset

315

- VoxPopuli (ES) - 108 hour subset

316

- Fisher - 141 hour subset

317

318

#### French (1.8k hours)

319

- Mozilla Common Voice (v12.0) - 708 hour subset

320

- Multilingual Librispeech (MLS FR) - 926 hour subset

321

- VoxPopuli (FR) - 165 hour subset

322

323

324

## Evaluation Dataset:

325

326

**Data Collection Method:** 

327

* Human 

328

329

**Labeling Method:** 

330

* Human 

331

332

Automatic Speech Recognition:

333

* [HuggingFace OpenASR Leaderboard evaluation sets](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard)

334

* [MLS](https://huggingface.co/datasets/facebook/multilingual_librispeech)

335

* [MCV] (https://commonvoice.mozilla.org/en/datasets)

336

337

Automatic Speech Translation:

338

* [FLEURS](https://huggingface.co/datasets/google/fleurs)

339

* [COVOST-v2](https://github.com/facebookresearch/covost)

340

* [mExpresso](https://huggingface.co/facebook/seamless-expressive#mexpresso-multilingual-expresso)

341

342

Timestamp Prediction:

343

* [Librispeech](https://www.openslr.org/12)

344

345

Hallucination Robustness:

346

* [MUSAN](https://www.openslr.org/17/) 48 hrs eval set

347

348

Noise Robustness:

349

* [Librispeech](https://www.openslr.org/12)

350

351

Model Fairness:

352

* [Casual Conversations Dataset](https://arxiv.org/pdf/2104.02821)

353

354

## Training

355

356

canary-180m-flash is trained using the NVIDIA NeMo framework [7] for a total of 219K steps with 2D bucketing [1] and optimal batch sizes set using OOMptimizer [8]. The model is trained on 32 NVIDIA A100 80GB GPUs.

357

The model can be trained using this [example script](https://github.com/NVIDIA/NeMo/blob/main/examples/asr/speech_multitask/speech_to_text_aed.py) and [base config](https://github.com/NVIDIA/NeMo/blob/main/examples/asr/conf/speech_multitask/fast-conformer_aed.yaml).

358

359

The tokenizers for these models were built using the text transcripts of the train set with this [script](https://github.com/NVIDIA/NeMo/blob/main/scripts/tokenizers/process_asr_text_tokenizer.py).

360

361

## Inference:

362

**Engine:** NVIDIA NeMo 

363

**Test Hardware :** 

364

* A6000 

365

* A100 

366

* V100 

367

368

## Performance

369

370

For ASR and AST experiments, predictions were generated using greedy decoding. Note that utterances shorter than 1 second are symmetrically zero-padded upto 1 second during evaluation.

371

372

### English ASR Performance (w/o PnC)

373

374

The ASR performance is measured with word error rate (WER), and we process the groundtruth and predicted text with [whisper-normalizer](https://pypi.org/project/whisper-normalizer/).

375

376

WER on [HuggingFace OpenASR leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard) evaluated with a batch size of 128:

377

378

| **Version** | **Model**     | **RTFx**   | **AMI**   | **GigaSpeech**   | **LS Clean**   | **LS Other**   | **Earnings22**   | **SPGISpech**   | **Tedlium**   | **Voxpopuli**   |

379

|:---------:|:-----------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|

380

| main | canary-180m-flash | 1233 | 14.86 | 10.51 | 1.87 | 3.83 | 13.33 | 2.26 | 3.98 | 6.35 |

381

382

#### Inference speed on different systems

383

We profiled inference speed on the OpenASR benchmark using the [real-time factor](https://github.com/NVIDIA/DeepLearningExamples/blob/master/Kaldi/SpeechRecognition/README.md#metrics) (RTFx) to quantify throughput.

384

385

386

|:-----------:|:-------------:|:------------:|:----------:|

387

| main | canary-180m-flash | NVIDIA A100 | 1233 |

388

| main | canary-180m-flash | NVIDIA H100 | 2041 |

389

390

391

392

### Multilingual ASR Performance

393

WER on [MLS](https://huggingface.co/datasets/facebook/multilingual_librispeech) test set:

394

395

| **Version** | **Model** | **De** | **Es** | **Fr** |

396

|:---------:|:-----------:|:------:|:------:|:------:|

397

| main | canary-180m-flash | 4.81 | 3.17 | 4.75 |

398

399

400

WER on [MCV-16.1](https://commonvoice.mozilla.org/en/datasets) test set:

401

| **Version** | **Model** | **En** | **De** | **Es** | **Fr** |

402

|:---------:|:-----------:|:------:|:------:|:------:|:------:|

403

| main | canary-180m-flash | 9.53 | 5.94 | 4.90 | 8.19 |

404

405

406

More details on evaluation can be found at [HuggingFace ASR Leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard)

407

408

### AST Performance

409

410

We evaluate AST performance with [BLEU score](https://lightning.ai/docs/torchmetrics/stable/text/sacre_bleu_score.html), and use native annotations with punctuation and capitalization in the datasets.

411

412

[FLEURS](https://huggingface.co/datasets/google/fleurs) test set:

413

414

BLEU score:

415

416

417

|:-----------:|:---------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|

418

| main | canary-180m-flash | 28.18 | 20.47 | 36.66 | 32.08 | 20.09 | 29.75 |

419

420

COMET score:

421

422

423

|:-----------:|:---------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|

424

| main | canary-180m-flash | 77.56 | 78.10 | 78.53 | 83.03 | 81.48 | 82.28 |

425

426

[COVOST-v2](https://github.com/facebookresearch/covost) test set:

427

428

BLEU score:

429

430

431

|:-----------:|:---------:|:----------:|:----------:|:----------:|

432

| main | canary-180m-flash | 35.61 | 39.84 | 38.57 |

433

434

COMET score:

435

436

437

|:-----------:|:---------:|:----------:|:----------:|:----------:|

438

| main | canary-180m-flash | 80.94 | 84.54 | 82.50 |

439

440

[mExpresso](https://huggingface.co/facebook/seamless-expressive#mexpresso-multilingual-expresso) test set:

441

442

BLEU score:

443

444

445

|:-----------:|:---------:|:----------:|:----------:|:----------:|

446

| main | canary-180m-flash | 21.60 | 33.45 | 25.96 |

447

448

COMET score:

449

450

451

|:-----------:|:---------:|:----------:|:----------:|:----------:|

452

| main | canary-180m-flash | 77.71 | 80.87 | 77.82 |

453

454

455

### Timestamp Prediction

456

F1-score on [Librispeech Test sets](https://www.openslr.org/12) at collar value of 200ms

457

458

459

|:-----------:|:---------:|:----------:|:----------:|

460

| main | canary-180m-flash | 93.48 | 91.38 |

461

462

463

### Hallucination Robustness

464

Number of characters per minute on [MUSAN](https://www.openslr.org/17) 48 hrs eval set

465

466

| **Version** | **Model** | **# of character per minute** |

467

|:-----------:|:---------:|:----------:|

468

| main | canary-180m-flash | 91.52 |

469

470

### Noise Robustness

471

WER on [Librispeech Test Clean](https://www.openslr.org/12) at different SNR (signal to noise ratio) levels of additive white noise

472

473

| **Version** | **Model** | **SNR 10** | **SNR 5** | **SNR 0** | **SNR -5** |

474

|:-----------:|:---------:|:----------:|:----------:|:----------:|:----------:|

475

| main | canary-180m-flash | 3.23 | 5.34 | 12.21 | 34.03 |

476

477

## Model Fairness Evaluation

478

479

As outlined in the paper "Towards Measuring Fairness in AI: the Casual Conversations Dataset" [9], we assessed the canary-180m-flash model for fairness. The model was evaluated on the CausalConversations-v1 dataset, and the results are reported as follows:

480

481

### Gender Bias:

482

483

| Gender | Male | Female | N/A | Other |

484

| :--- | :--- | :--- | :--- | :--- |

485

| Num utterances | 19325 | 24532 | 926 | 33 |

486

| % WER | 16.92 | 14.01 | 20.01 | 25.04 |

487

488

### Age Bias:

489

490

| Age Group | (18-30) | (31-45) | (46-85) | (1-100) |

491

| :--- | :--- | :--- | :--- | :--- |

492

| Num utterances | 15956 | 14585 | 13349 | 43890 |

493

| % WER | 14.95 | 15.36 | 15.65 | 15.29 |

494

495

(Error rates for fairness evaluation are determined by normalizing both the reference and predicted text, similar to the methods used in the evaluations found at https://github.com/huggingface/open_asr_leaderboard.)

496

497

## License/Terms of Use:

498

canary-180m-flash is released under the CC-BY-4.0 license. By using this model, you are agreeing to the [terms and conditions](https://choosealicense.com/licenses/cc-by-4.0/) of the license. <br>

499

500

## References:

501

502

[1] [Training and Inference Efficiency of Encoder-Decoder Speech Models](https://arxiv.org/pdf/2503.05931)

503

504

[2] [Less is More: Accurate Speech Recognition & Translation without Web-Scale Data](https://www.isca-archive.org/interspeech_2024/puvvada24_interspeech.pdf) <br>

505

506

[3] [Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=10389701)

507

508

[4] [Attention is All You Need](https://arxiv.org/abs/1706.03762)

509

510

[5] [Unified Model for Code-Switching Speech Recognition and Language Identification Based on Concatenated Tokenizer](https://aclanthology.org/2023.calcs-1.7.pdf)

511

512

[6] [Google Sentencepiece Tokenizer](https://github.com/google/sentencepiece)

513

514

[7] [NVIDIA NeMo Framework](https://github.com/NVIDIA/NeMo)

515

516

[8] [EMMeTT: Efficient Multimodal Machine Translation Training](https://arxiv.org/abs/2409.13523)

517

518

[9] [Towards Measuring Fairness in AI: the Casual Conversations Dataset](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9634168)

519

520

521

## Ethical Considerations:

522

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications.  When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

523

Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).

524