New paper: Multi-Faceted Interactivity Alignment in Full-Duplex Speech Models We use RL to post-train speech models (Moshi and PersonaPlex) to talk more like a human: to know when to respond, when to wait, and when to nod along with “yeah”s and “okay”s when listening.

Full-duplex speech models like Moshi and PersonaPlex can theoretically sound more natural than their more standard turn-based counterparts, but in practice they still have issues like staying silent when they shouldn't, talking at the wrong time, and not backchanneling.

A big reason for that are the token-level losses used in supervised learning. Staying silent instead of backchanneling costs little in cross-entropy loss, but is noticeable. Post-training with RL allows us to express sequence-level goals like “backchannel every now and then”.

Paper: Blog: Models:

We applied our method to both Moshi and PersonaPlex (from @nvidia) and saw improvements across the board. We evaluated on both Full-Duplex-Bench v1, which feeds pre-recorded audio to the model, and v2, where the model has a live multi-turn conversation.

How are you guys so fast at implementing things that we have barely got up and running! Awesome work BTW 🙂

Thank you and great job on PersonaPlex!

Incredible work from Kyut ai as always!!

Speech models now converse more naturally

Wooow, amazing. I read the Moshi paper and was really impressed. I'll definitely be reading this one as well!

“ShMedium” - legendary

Amazing

insane work

Being Conversational is the first priority

@beknabdik

Well done ✨🎉👏🏽

The hard part sits under the reward model: 'when to backchannel' has no single ground truth. Two annotators will mark the same exchange differently and both be right. Whoever solves consistency of judgement across the labeling workforce owns this benchmark.