imedennikov commited on
Commit
9f01a75
·
verified ·
1 Parent(s): 78c0cc2

Upload ASR_INTEGRATION_GUIDE.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. ASR_INTEGRATION_GUIDE.md +29 -20
ASR_INTEGRATION_GUIDE.md CHANGED
@@ -11,8 +11,9 @@ This guide shows two supported streaming ASR pairings:
11
  - ASR option 2: [`nvidia/nemotron-3.5-asr-streaming-0.6b`](https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b)
12
  - Integration: NeMo Speech's [`speech_to_text_multitalker_streaming_infer.py`](https://github.com/NVIDIA-NeMo/Speech/blob/main/examples/asr/asr_cache_aware_streaming/speech_to_text_multitalker_streaming_infer.py)
13
 
14
- This is a coupled streaming pipeline. Nemotron-3-diarization is a speaker diarization model
15
- and it produces speaker activity for each frame, and the ASR stage runs one stream per speaker.
 
16
 
17
  ## 1. Install NeMo Speech
18
 
@@ -42,17 +43,21 @@ export HF_TOKEN="your-hugging-face-token"
42
 
43
  Do not put the token in source code or commit it to a repository.
44
 
45
- Download the diarization model locally and pass the downloaded `.nemo` file to
46
- the NeMo example script:
 
 
 
 
 
47
 
48
  ```bash
49
  hf download nvidia/Nemotron-3-Diarization \
50
  --local-dir /path/to/Nemotron-3-Diarization
51
  ```
52
 
53
- The NeMo example script accepts a local `.nemo` checkpoint path for
54
- `diar_model`. Do not pass `diar_model=nvidia/Nemotron-3-Diarization`; that Hugging
55
- Face repository ID is not a supported `diar_model` value in the example script.
56
 
57
  ## 2. Prepare the audio
58
 
@@ -65,8 +70,7 @@ ffmpeg -i input.mp3 -ac 1 -ar 16000 -c:a pcm_s16le conversation.wav
65
 
66
  ## 3. Choose the ASR model
67
 
68
- Use one of the following ASR configurations with the local Nemotron-3-Diarization
69
- checkpoint.
70
 
71
  ### Option 1: Multitalker Parakeet
72
 
@@ -79,7 +83,7 @@ From the NeMo Speech repository root, run:
79
  ```bash
80
  python examples/asr/asr_cache_aware_streaming/speech_to_text_multitalker_streaming_infer.py \
81
  asr_model=nvidia/multitalker-parakeet-streaming-0.6b-v1 \
82
- diar_model=/path/to/Nemotron-3-Diarization/Nemotron-3-Diarization.nemo \
83
  audio_file=/absolute/path/to/conversation.wav \
84
  max_num_of_spks=8 \
85
  single_speaker_mode=false \
@@ -99,8 +103,9 @@ python examples/asr/asr_cache_aware_streaming/speech_to_text_multitalker_streami
99
  [`nvidia/nemotron-3.5-asr-streaming-0.6b`](https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b)
100
  is a regular single-speaker streaming ASR model. In this speaker-attributed
101
  pipeline, it can use masked speaker activity to recognize overlapped speech to a
102
- certain degree. It supports 40 language-locales and should be used with
103
- `masked_asr=true`.
 
104
 
105
  Set `target_lang` to the input language-locale, or use `target_lang=auto` when
106
  the model should infer the language prompt automatically.
@@ -108,7 +113,7 @@ the model should infer the language prompt automatically.
108
  ```bash
109
  python examples/asr/asr_cache_aware_streaming/speech_to_text_multitalker_streaming_infer.py \
110
  asr_model=nvidia/nemotron-3.5-asr-streaming-0.6b \
111
- diar_model=/path/to/Nemotron-3-Diarization/Nemotron-3-Diarization.nemo \
112
  audio_file=/absolute/path/to/conversation.wav \
113
  max_num_of_spks=8 \
114
  single_speaker_mode=false \
@@ -117,16 +122,17 @@ python examples/asr/asr_cache_aware_streaming/speech_to_text_multitalker_streami
117
  cache_gating=true \
118
  binary_diar_preds=true \
119
  att_context_size='[56,13]' \
120
- fifo_len=200 \
121
  spkcache_update_period=222 \
122
  target_lang=auto \
123
  generate_realtime_scripts=false \
124
  output_path=./speaker_attributed_output.json
125
  ```
126
 
127
- The script prints streaming hypotheses and writes the final result to
128
- `speaker_attributed_output.json` in NeMo's SegLST format. Each finalized item
129
- contains its text, time range, and anonymous speaker label.
 
130
 
131
  The important settings are:
132
 
@@ -138,8 +144,9 @@ The important settings are:
138
  | `masked_asr=true` | Use this with Nemotron 3.5 ASR, which uses diarization activity to mask each speaker stream. |
139
  | `att_context_size='[70,13]'` | Recommended context for Multitalker Parakeet. |
140
  | `att_context_size='[56,13]'` | Recommended context for Nemotron 3.5 ASR. |
141
- | `cache_gating=true` | Stabilizes speaker-cache updates during streaming. |
142
- | `spkcache_update_period=222` | Refreshes the speaker cache frequently enough for the long, eight-speaker sessions used by this example. Tune this for other workloads. |
 
143
 
144
  `max_num_of_spks` is an upper bound, not a promise that every configured speaker
145
  will appear. The diarization model discovers active speaker identities from the
@@ -183,7 +190,9 @@ The service structure is:
183
  diarization activity state to the client.
184
  5. Reset the client's session when the recording stops or the websocket closes.
185
 
186
- The core streaming loop looks like this:
 
 
187
 
188
  ```python
189
  import os
 
11
  - ASR option 2: [`nvidia/nemotron-3.5-asr-streaming-0.6b`](https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b)
12
  - Integration: NeMo Speech's [`speech_to_text_multitalker_streaming_infer.py`](https://github.com/NVIDIA-NeMo/Speech/blob/main/examples/asr/asr_cache_aware_streaming/speech_to_text_multitalker_streaming_infer.py)
13
 
14
+ This is a coupled streaming pipeline. Nemotron-3-Diarization produces
15
+ frame-level speaker activity, while the ASR stage maintains a separate
16
+ transcription stream for each detected speaker.
17
 
18
  ## 1. Install NeMo Speech
19
 
 
43
 
44
  Do not put the token in source code or commit it to a repository.
45
 
46
+ The NeMo example script accepts the Hugging Face repository ID directly:
47
+
48
+ ```bash
49
+ diar_model=nvidia/Nemotron-3-Diarization
50
+ ```
51
+
52
+ To keep a local copy of the checkpoint instead, download it with:
53
 
54
  ```bash
55
  hf download nvidia/Nemotron-3-Diarization \
56
  --local-dir /path/to/Nemotron-3-Diarization
57
  ```
58
 
59
+ Then replace the repository ID in the commands below with
60
+ `/path/to/Nemotron-3-Diarization/Nemotron-3-Diarization.nemo`.
 
61
 
62
  ## 2. Prepare the audio
63
 
 
70
 
71
  ## 3. Choose the ASR model
72
 
73
+ Use one of the following ASR configurations with Nemotron-3-Diarization.
 
74
 
75
  ### Option 1: Multitalker Parakeet
76
 
 
83
  ```bash
84
  python examples/asr/asr_cache_aware_streaming/speech_to_text_multitalker_streaming_infer.py \
85
  asr_model=nvidia/multitalker-parakeet-streaming-0.6b-v1 \
86
+ diar_model=nvidia/Nemotron-3-Diarization \
87
  audio_file=/absolute/path/to/conversation.wav \
88
  max_num_of_spks=8 \
89
  single_speaker_mode=false \
 
103
  [`nvidia/nemotron-3.5-asr-streaming-0.6b`](https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b)
104
  is a regular single-speaker streaming ASR model. In this speaker-attributed
105
  pipeline, it can use masked speaker activity to recognize overlapped speech to a
106
+ certain degree. It supports 32 language-locales out of the box; its tokenizer
107
+ supports eight additional adaptation-ready locales that require fine-tuning.
108
+ Use it with `masked_asr=true`.
109
 
110
  Set `target_lang` to the input language-locale, or use `target_lang=auto` when
111
  the model should infer the language prompt automatically.
 
113
  ```bash
114
  python examples/asr/asr_cache_aware_streaming/speech_to_text_multitalker_streaming_infer.py \
115
  asr_model=nvidia/nemotron-3.5-asr-streaming-0.6b \
116
+ diar_model=nvidia/Nemotron-3-Diarization \
117
  audio_file=/absolute/path/to/conversation.wav \
118
  max_num_of_spks=8 \
119
  single_speaker_mode=false \
 
122
  cache_gating=true \
123
  binary_diar_preds=true \
124
  att_context_size='[56,13]' \
125
+ fifo_len=264 \
126
  spkcache_update_period=222 \
127
  target_lang=auto \
128
  generate_realtime_scripts=false \
129
  output_path=./speaker_attributed_output.json
130
  ```
131
 
132
+ The script writes the final result to `speaker_attributed_output.json` in
133
+ NeMo's SegLST format. Each finalized item contains its text, time range, and
134
+ anonymous speaker label. Set `generate_realtime_scripts=true` to also print
135
+ streaming hypotheses.
136
 
137
  The important settings are:
138
 
 
144
  | `masked_asr=true` | Use this with Nemotron 3.5 ASR, which uses diarization activity to mask each speaker stream. |
145
  | `att_context_size='[70,13]'` | Recommended context for Multitalker Parakeet. |
146
  | `att_context_size='[56,13]'` | Recommended context for Nemotron 3.5 ASR. |
147
+ | `cache_gating=true` | Runs speaker-specific ASR only for speakers detected as active in the recent diarization window, reducing unnecessary ASR computation. |
148
+ | `fifo_len=264` | Maximum number of recent 80 ms diarization encoder frames retained in the FIFO queue. |
149
+ | `spkcache_update_period=222` | Number of the FIFO queue's oldest 80 ms frames transferred during each speaker-cache update; the effective value is constrained by the chunk and FIFO geometry. |
150
 
151
  `max_num_of_spks` is an upper bound, not a promise that every configured speaker
152
  will appear. The diarization model discovers active speaker identities from the
 
190
  diarization activity state to the client.
191
  5. Reset the client's session when the recording stops or the websocket closes.
192
 
193
+ The illustrative streaming loop below is configured specifically for Option 1,
194
+ Multitalker Parakeet. Option 2 requires its corresponding model-specific
195
+ settings.
196
 
197
  ```python
198
  import os