Voice Activity Detection
NeMo
Safetensors
GGUF
Transformers
nemotron3_diarization
audio-frame-classification
speaker-diarization
streaming-sortformer
speaker-tagging
Instructions to use nvidia/Nemotron-3-Diarization with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- NeMo
How to use nvidia/Nemotron-3-Diarization with NeMo:
# tag did not correspond to a valid NeMo domain.
- Transformers
How to use nvidia/Nemotron-3-Diarization with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForAudioFrameClassification processor = AutoProcessor.from_pretrained("nvidia/Nemotron-3-Diarization") model = AutoModelForAudioFrameClassification.from_pretrained("nvidia/Nemotron-3-Diarization", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Upload ASR_INTEGRATION_GUIDE.md with huggingface_hub
Browse files- ASR_INTEGRATION_GUIDE.md +29 -20
ASR_INTEGRATION_GUIDE.md
CHANGED
|
@@ -11,8 +11,9 @@ This guide shows two supported streaming ASR pairings:
|
|
| 11 |
- ASR option 2: [`nvidia/nemotron-3.5-asr-streaming-0.6b`](https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b)
|
| 12 |
- Integration: NeMo Speech's [`speech_to_text_multitalker_streaming_infer.py`](https://github.com/NVIDIA-NeMo/Speech/blob/main/examples/asr/asr_cache_aware_streaming/speech_to_text_multitalker_streaming_infer.py)
|
| 13 |
|
| 14 |
-
This is a coupled streaming pipeline. Nemotron-3-
|
| 15 |
-
|
|
|
|
| 16 |
|
| 17 |
## 1. Install NeMo Speech
|
| 18 |
|
|
@@ -42,17 +43,21 @@ export HF_TOKEN="your-hugging-face-token"
|
|
| 42 |
|
| 43 |
Do not put the token in source code or commit it to a repository.
|
| 44 |
|
| 45 |
-
|
| 46 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 47 |
|
| 48 |
```bash
|
| 49 |
hf download nvidia/Nemotron-3-Diarization \
|
| 50 |
--local-dir /path/to/Nemotron-3-Diarization
|
| 51 |
```
|
| 52 |
|
| 53 |
-
|
| 54 |
-
`
|
| 55 |
-
Face repository ID is not a supported `diar_model` value in the example script.
|
| 56 |
|
| 57 |
## 2. Prepare the audio
|
| 58 |
|
|
@@ -65,8 +70,7 @@ ffmpeg -i input.mp3 -ac 1 -ar 16000 -c:a pcm_s16le conversation.wav
|
|
| 65 |
|
| 66 |
## 3. Choose the ASR model
|
| 67 |
|
| 68 |
-
Use one of the following ASR configurations with
|
| 69 |
-
checkpoint.
|
| 70 |
|
| 71 |
### Option 1: Multitalker Parakeet
|
| 72 |
|
|
@@ -79,7 +83,7 @@ From the NeMo Speech repository root, run:
|
|
| 79 |
```bash
|
| 80 |
python examples/asr/asr_cache_aware_streaming/speech_to_text_multitalker_streaming_infer.py \
|
| 81 |
asr_model=nvidia/multitalker-parakeet-streaming-0.6b-v1 \
|
| 82 |
-
diar_model=/
|
| 83 |
audio_file=/absolute/path/to/conversation.wav \
|
| 84 |
max_num_of_spks=8 \
|
| 85 |
single_speaker_mode=false \
|
|
@@ -99,8 +103,9 @@ python examples/asr/asr_cache_aware_streaming/speech_to_text_multitalker_streami
|
|
| 99 |
[`nvidia/nemotron-3.5-asr-streaming-0.6b`](https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b)
|
| 100 |
is a regular single-speaker streaming ASR model. In this speaker-attributed
|
| 101 |
pipeline, it can use masked speaker activity to recognize overlapped speech to a
|
| 102 |
-
certain degree. It supports
|
| 103 |
-
|
|
|
|
| 104 |
|
| 105 |
Set `target_lang` to the input language-locale, or use `target_lang=auto` when
|
| 106 |
the model should infer the language prompt automatically.
|
|
@@ -108,7 +113,7 @@ the model should infer the language prompt automatically.
|
|
| 108 |
```bash
|
| 109 |
python examples/asr/asr_cache_aware_streaming/speech_to_text_multitalker_streaming_infer.py \
|
| 110 |
asr_model=nvidia/nemotron-3.5-asr-streaming-0.6b \
|
| 111 |
-
diar_model=/
|
| 112 |
audio_file=/absolute/path/to/conversation.wav \
|
| 113 |
max_num_of_spks=8 \
|
| 114 |
single_speaker_mode=false \
|
|
@@ -117,16 +122,17 @@ python examples/asr/asr_cache_aware_streaming/speech_to_text_multitalker_streami
|
|
| 117 |
cache_gating=true \
|
| 118 |
binary_diar_preds=true \
|
| 119 |
att_context_size='[56,13]' \
|
| 120 |
-
fifo_len=
|
| 121 |
spkcache_update_period=222 \
|
| 122 |
target_lang=auto \
|
| 123 |
generate_realtime_scripts=false \
|
| 124 |
output_path=./speaker_attributed_output.json
|
| 125 |
```
|
| 126 |
|
| 127 |
-
The script
|
| 128 |
-
|
| 129 |
-
|
|
|
|
| 130 |
|
| 131 |
The important settings are:
|
| 132 |
|
|
@@ -138,8 +144,9 @@ The important settings are:
|
|
| 138 |
| `masked_asr=true` | Use this with Nemotron 3.5 ASR, which uses diarization activity to mask each speaker stream. |
|
| 139 |
| `att_context_size='[70,13]'` | Recommended context for Multitalker Parakeet. |
|
| 140 |
| `att_context_size='[56,13]'` | Recommended context for Nemotron 3.5 ASR. |
|
| 141 |
-
| `cache_gating=true` |
|
| 142 |
-
| `
|
|
|
|
| 143 |
|
| 144 |
`max_num_of_spks` is an upper bound, not a promise that every configured speaker
|
| 145 |
will appear. The diarization model discovers active speaker identities from the
|
|
@@ -183,7 +190,9 @@ The service structure is:
|
|
| 183 |
diarization activity state to the client.
|
| 184 |
5. Reset the client's session when the recording stops or the websocket closes.
|
| 185 |
|
| 186 |
-
The
|
|
|
|
|
|
|
| 187 |
|
| 188 |
```python
|
| 189 |
import os
|
|
|
|
| 11 |
- ASR option 2: [`nvidia/nemotron-3.5-asr-streaming-0.6b`](https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b)
|
| 12 |
- Integration: NeMo Speech's [`speech_to_text_multitalker_streaming_infer.py`](https://github.com/NVIDIA-NeMo/Speech/blob/main/examples/asr/asr_cache_aware_streaming/speech_to_text_multitalker_streaming_infer.py)
|
| 13 |
|
| 14 |
+
This is a coupled streaming pipeline. Nemotron-3-Diarization produces
|
| 15 |
+
frame-level speaker activity, while the ASR stage maintains a separate
|
| 16 |
+
transcription stream for each detected speaker.
|
| 17 |
|
| 18 |
## 1. Install NeMo Speech
|
| 19 |
|
|
|
|
| 43 |
|
| 44 |
Do not put the token in source code or commit it to a repository.
|
| 45 |
|
| 46 |
+
The NeMo example script accepts the Hugging Face repository ID directly:
|
| 47 |
+
|
| 48 |
+
```bash
|
| 49 |
+
diar_model=nvidia/Nemotron-3-Diarization
|
| 50 |
+
```
|
| 51 |
+
|
| 52 |
+
To keep a local copy of the checkpoint instead, download it with:
|
| 53 |
|
| 54 |
```bash
|
| 55 |
hf download nvidia/Nemotron-3-Diarization \
|
| 56 |
--local-dir /path/to/Nemotron-3-Diarization
|
| 57 |
```
|
| 58 |
|
| 59 |
+
Then replace the repository ID in the commands below with
|
| 60 |
+
`/path/to/Nemotron-3-Diarization/Nemotron-3-Diarization.nemo`.
|
|
|
|
| 61 |
|
| 62 |
## 2. Prepare the audio
|
| 63 |
|
|
|
|
| 70 |
|
| 71 |
## 3. Choose the ASR model
|
| 72 |
|
| 73 |
+
Use one of the following ASR configurations with Nemotron-3-Diarization.
|
|
|
|
| 74 |
|
| 75 |
### Option 1: Multitalker Parakeet
|
| 76 |
|
|
|
|
| 83 |
```bash
|
| 84 |
python examples/asr/asr_cache_aware_streaming/speech_to_text_multitalker_streaming_infer.py \
|
| 85 |
asr_model=nvidia/multitalker-parakeet-streaming-0.6b-v1 \
|
| 86 |
+
diar_model=nvidia/Nemotron-3-Diarization \
|
| 87 |
audio_file=/absolute/path/to/conversation.wav \
|
| 88 |
max_num_of_spks=8 \
|
| 89 |
single_speaker_mode=false \
|
|
|
|
| 103 |
[`nvidia/nemotron-3.5-asr-streaming-0.6b`](https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b)
|
| 104 |
is a regular single-speaker streaming ASR model. In this speaker-attributed
|
| 105 |
pipeline, it can use masked speaker activity to recognize overlapped speech to a
|
| 106 |
+
certain degree. It supports 32 language-locales out of the box; its tokenizer
|
| 107 |
+
supports eight additional adaptation-ready locales that require fine-tuning.
|
| 108 |
+
Use it with `masked_asr=true`.
|
| 109 |
|
| 110 |
Set `target_lang` to the input language-locale, or use `target_lang=auto` when
|
| 111 |
the model should infer the language prompt automatically.
|
|
|
|
| 113 |
```bash
|
| 114 |
python examples/asr/asr_cache_aware_streaming/speech_to_text_multitalker_streaming_infer.py \
|
| 115 |
asr_model=nvidia/nemotron-3.5-asr-streaming-0.6b \
|
| 116 |
+
diar_model=nvidia/Nemotron-3-Diarization \
|
| 117 |
audio_file=/absolute/path/to/conversation.wav \
|
| 118 |
max_num_of_spks=8 \
|
| 119 |
single_speaker_mode=false \
|
|
|
|
| 122 |
cache_gating=true \
|
| 123 |
binary_diar_preds=true \
|
| 124 |
att_context_size='[56,13]' \
|
| 125 |
+
fifo_len=264 \
|
| 126 |
spkcache_update_period=222 \
|
| 127 |
target_lang=auto \
|
| 128 |
generate_realtime_scripts=false \
|
| 129 |
output_path=./speaker_attributed_output.json
|
| 130 |
```
|
| 131 |
|
| 132 |
+
The script writes the final result to `speaker_attributed_output.json` in
|
| 133 |
+
NeMo's SegLST format. Each finalized item contains its text, time range, and
|
| 134 |
+
anonymous speaker label. Set `generate_realtime_scripts=true` to also print
|
| 135 |
+
streaming hypotheses.
|
| 136 |
|
| 137 |
The important settings are:
|
| 138 |
|
|
|
|
| 144 |
| `masked_asr=true` | Use this with Nemotron 3.5 ASR, which uses diarization activity to mask each speaker stream. |
|
| 145 |
| `att_context_size='[70,13]'` | Recommended context for Multitalker Parakeet. |
|
| 146 |
| `att_context_size='[56,13]'` | Recommended context for Nemotron 3.5 ASR. |
|
| 147 |
+
| `cache_gating=true` | Runs speaker-specific ASR only for speakers detected as active in the recent diarization window, reducing unnecessary ASR computation. |
|
| 148 |
+
| `fifo_len=264` | Maximum number of recent 80 ms diarization encoder frames retained in the FIFO queue. |
|
| 149 |
+
| `spkcache_update_period=222` | Number of the FIFO queue's oldest 80 ms frames transferred during each speaker-cache update; the effective value is constrained by the chunk and FIFO geometry. |
|
| 150 |
|
| 151 |
`max_num_of_spks` is an upper bound, not a promise that every configured speaker
|
| 152 |
will appear. The diarization model discovers active speaker identities from the
|
|
|
|
| 190 |
diarization activity state to the client.
|
| 191 |
5. Reset the client's session when the recording stops or the websocket closes.
|
| 192 |
|
| 193 |
+
The illustrative streaming loop below is configured specifically for Option 1,
|
| 194 |
+
Multitalker Parakeet. Option 2 requires its corresponding model-specific
|
| 195 |
+
settings.
|
| 196 |
|
| 197 |
```python
|
| 198 |
import os
|