Загрузка видео...
Не удалось загрузить видео
New week, new tiny model for Pipecat. Pipecat Voice Reader listens to how something is said: tone of voice, energy, laughter, speaking rate and potential sarcasm. It's a fine-tuned version of LAION’s affect-trained Whisper encoder, in two sizes (base or tiny), and runs on CPU in well under 50... show more
16,466 просмотров • 14 дней назад •via X (Twitter)
Комментарии: 25

The "great, just great" detector, finally 😄 How does sarcasm hold up across languages? Spanish sarcasm is basically a national sport.

Being a British, game on! 😅 Right now, this is English only. Since it’s so easy and fast to fine-tune these weights, it simply comes down to well annotated data.

British vs Spanish sarcasm is a benchmark I'd pay to watch 😄 And yes, labeling is the hard part. Humans disagree on half the sarcastic lines too.

@JonPTaylor Wonder if sarcasm is universal or if it differs between languages

Differs a lot. In Spanish it often rides on a stretched vowel ("claaaro"), in English more on a flat, deadpan pitch. Same intent, different melody. A model trained on one will miss it in the other.

Claaaaaro 😁 I remember this research from Hume finding the sounds people use to express emotions are largely universal (with some cultural differences)

Makes sense. The sounds travel (a laugh, a sigh), the melody on the words doesn't. That's why non-verbal tags port across languages and prosody has to be learned per language.

this is the right shape: a tiny model reading how it's said beside the one deciding what to say. ours does a read in ~0.2s per turn. next to an agent that's nothing, inside a live conversation it's the difference between answering the words and answering the person

Tone on the CPU is a great fit for StarStranded, my Godot game where NPCs run on an offline local model. Voice input already uses whisper.cpp base.en on the CPU since the GPU is busy with the NPC model. Does the tiny Voice Reader run alongside it on the same audio chunks?

It could do, yes. The purpose of keeping these models small is on-device consideration. We have an Unreal engine demo in the Pipecat org that demos an approach - I'll have to update it to use this model too!

An Unreal demo is a good sign it can live inside a game loop. My game is in Godot, so I'd wire it in as a sidecar next to llama-server and whisper.cpp. Does the demo run the model in-process, or as a separate local service the engine talks to?

Is it only for emotions or it can detect other aspects as well ?

I like keeping the voice reading beside the text reading. If the words sound positive but the delivery doesn’t, that disagreement seems worth preserving rather than collapsing it into one label.

That's a great innovation. Besides the text, the context and the mood of the speaker is super important for AI to decide the next step. Inspiring!

Cascaded pipelines often add more latency than speech-to-speech. The edge depends more on your latency budget than the architecture.

This is getting good

Tiny models are terribly underrated.

tiny model catching sarcasm from tone

Sarcasm is context-dependent, so uncertainty is worth preserving in the output.

under 50ms on cpu is what makes this usable. the part i like is you get the tone reading as something you can log per turn, so when a call goes sideways you can actually see the turn where the user's energy dropped. with speech to speech that's buried inside the model

finally something that can tell when i say 'no worries' and mean worries

So many worries. We also need to train a “passive aggressive” head on all these models too 😅

a 'per my last email' detector would sell itself

tone is the thing almost every voice eval skips right now, so a small model for it is great. does it hold up when people switch languages mid sentence, like hindi and english together?

we are in the golden age of voice tech for sure
