NVIDIA released a fast speaker-labeling model today: Nemotron 3 Diarization. Like all of the Nemotron family from @NVIDIA_AI, this is an open weights model, so you can grab the weights from Hugging Face and use the model in the cloud or on your local device. Nemotron 3 Diarization is... a useful building block for voice agents and realtime AI applications that need to handle speech from multiple people at the same time. It handles overlapping speech well, can identify up to eight speakers, and pairs with any transcription model you're using in your multi-modal pipelines. Jon Taylor built a demo (code below) and recorded a technical deep-dive showing how to use this diarization model together with a speech-to-text model. Jon's code also uses Jev for super-accurate confirmation of critical voice input in a noisy/multi-speaker environment.show more

NVIDIA Nemotron 3 Diarization model card and weights: Jon's demo code: Models used in the demo: - NVIDIA Nemotron 3 Diarization - NVIDIA Nemotron 3.5 ASR - Pipecat Smart Turn - Pipecat PhoneLLM - Typesafe Jev - NVIDIA Magpie TTS

The NVIDIA launch post here on X, with links to model benchmarks:

@nvidia_ai @huggingface very cool!

@nvidia_ai @huggingface Already working on getting this in my app for audio recordings 🔥

@nvidia_ai @huggingface This is fantastic. Very fantastic

@nvidia_ai @huggingface On phone interviews diarization never paid rent. Our false barge-ins came from a TV in the room, not a second speaker on the line.

Agreed. Phone use cases don't need diarization. A good primary speaker isolation model (a noise reduction model that removes all background noise including background speech) is a better fit. Diarization is important for use cases where you need the agent to "hear" multiple speakers, and you need the agent to know/track the different speaker identities.

@k0nOO @nvidia_ai @huggingface Are there some good noise reduction model out there which we can integrate in pipecat pipeline. Are there some code examples for that ?

@k0nOO @nvidia_ai @huggingface We usually recommend @krispHQ. It’s a commercial model so it isn’t free. But it’s available through Pipecat Cloud with 10,000 free minutes per month.

@nvidia_ai @huggingface this is something really awesome! looking for this for long

@nvidia_ai @huggingface Getting every word right and assigning the promise to the wrong person is how a meeting bot invents volunteers.

@nvidia_ai @huggingface Is give this to @GMHikaru and then no more mouse slips only “takes takes takes”

@nvidia_ai @huggingface Pairing it with whatever transcriber you already run is the practical move. Jev only stepping in to confirm the noisy bit is a sane split.