Voice agents still don’t understand who’s speaking to them. That’s a huge gap compared with humans, hidden by all the “phone-call” demos. But that changes today! NVIDIA is open-sourcing Nemotron 3 Diarization: a model that can reliably track speakers in live conversations, under a commercial-friendly license! In my tests,... the quality is really good with one-second speech chunks. So we can use it for voice agents! I tested it with Reachy Mini and speech-to-speech running on a DGX Spark. It’s super fun to see the robot notice a new voice, ask for a name, and remember it. The model has day-zero integration with Transformers! Kudos to the NVIDIA team for shipping useful tools for the whole community!show more

This is an incredible development. That NVIDIA is sharing this technology makes it even better. We're headed toward the cool future, aren't we?

Totally! Great future ahead

Voice agents don't fail on the LLM half — they fail when they can't tell who said what. Diarization as open infra is the boring layer that makes multi-party agents usable.

how does it hold up on cross talk, overlapping speakers is where one second chunks usually fall apart

speaker drift is what actually breaks those demos in prod

i like this for agents; transcripts need speaker labels when calls overlap.

Running the open Nemotron diarization locally on a Spark is the part that matters. Open weights only help if inference stays next to the audio, not on someone else's API.

We are already working on this in production at

Layperson here: Why can Alexa differenciate betwen my wife, myself, and my kids? Even when there are other people in the house around the same age of the respective person. If the voice doesn't match anyone from our normal every day house hold... it doesn't respond to them directly but instead more generally.

this is the missing piece for voice agents that actually work in a room. the phone-call demos hide the fact that most agents panic the second two people talk at once

walk-mode agent should only call for a real yes/no decision

This matters a lot on real phone calls, where a spouse or a colleague often picks up halfway through. Knowing the speaker changed lets the agent stop and re-check identity before it shares account details.

speaker overlap is the hard part for live calls. does Nemotron 3 handle crosstalk reliably or does it still need clean turn-taking?

Asking for the name is the smart bit. Diarization alone only gives you speaker 0 and speaker 1

@huggingface damn i wish i had my reachy with me rn 😭 good stuff 🔥

This is huge for voice agents in shared spaces.

The bottleneck in voice agents isn't more parameters; it's acoustic telemetry latency. True diarization requires extracting raw communicative primitives at the edge, not routing conversational audio to cloud servers. Sovereignty starts with local acoustic logic. ♟️

one-second chunks should make the live handoff a lot less janky

Backup (no product): Phone demos hide speaker confusion hard. Feels like this closes a real gap vs humans.

Voice AI tool to check out later