正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Kyutai released their Streaming Text to Speech model, ~2B param model, ultra low latency (220ms), CC-BY-4.0 license 🔥 Trained on 2.5 Million Hours of audio, it can serve up to 32 users w/ less than 350ms latency on a SINGLE L40 🤯 Incredible release by kyutai folks, go check... show more

Vaibhav (VB) Srivastav

55,570 subscribers

93,586 次观看 • 1 年前 •via X (Twitter)

科学技术新闻政治教育

Anya Rossi• Live Now

Private livecam show

6 条评论

Vaibhav (VB) Srivastav 的头像

Vaibhav (VB) Srivastav1 年前

Check out their models here:

Aakash 的头像

Aakash1 年前

"Trained on 2.5 Million Hours of audio, it can serve up to 32 users w/ less than 350ms latency on a SINGLE L40" can we get more of this benchmark

KD 的头像

KD1 年前

These are some of the same guys who run a really amazing YT channel about CS btw:

ZAZO 的头像

ZAZO1 年前

that’s the best thing happened in 2025 🔥🔥🔥🔥🔥🔥🔥🔥🔥

Bui Dinh Ngoc 的头像

Bui Dinh Ngoc1 年前

This is game-changing for accessibility tools. I've been waiting for low-latency TTS that doesn't break the bank or require proprietary licenses.

Carlos DP 的头像

Carlos DP1 年前

SUCH a solid demo lol, S tier

相关视频

Kyutai TTS and Unmute are now open source! The text-to-speech is natural, customizable, and fast: it can serve 32 users with a 350ms latency on a single L40S. Try it out and get started on the project page:

Kyutai TTS and Unmute are now open source! The text-to-speech is natural, customizable, and fast: it can serve 32 users with a 350ms latency on a single L40S. Try it out and get started on the project page:

kyutai

172,287 次观看 • 1 年前

LETS GOOO! FishSpeech v1.5 - multilingual, zero-shot instant voice cloning, low-latency, open text to speech model 🔥 > Only 500M params > Trained on 1 MILLION hours of audio > Supports 13 languages > Low-latency ( Open model - checkpoints on the hub 🤗 > Best part: Ranks #2 on TTS Arena (preliminary results) Kudos to Fish Audio - truly leading the way for open text to speech research! ⚡

LETS GOOO! FishSpeech v1.5 - multilingual, zero-shot instant voice cloning, low-latency, open text to speech model 🔥 > Only 500M params > Trained on 1 MILLION hours of audio > Supports 13 languages > Low-latency ( Open model - checkpoints on the hub 🤗 > Best part: Ranks #2 on TTS Arena (preliminary results) Kudos to Fish Audio - truly leading the way for open text to speech research! ⚡

Vaibhav (VB) Srivastav

49,150 次观看 • 1 年前

Wow! New Speech to Speech model - Fish Agent v0.1 3B by Fish Audio 🔥 > Trained on 700K hours of multilingual audio > Continue-pretrained version of Qwen-2.5-3B-Instruct for 200B audio & text tokens > Zero-shot voice cloning > Text + audio input/ Audio output > Ultra-fast inference w/ 200ms TTFA > Models on the Hub & Finetuning code on its way! 🚀 What an amazing time to be alive 🤗

Wow! New Speech to Speech model - Fish Agent v0.1 3B by Fish Audio 🔥 > Trained on 700K hours of multilingual audio > Continue-pretrained version of Qwen-2.5-3B-Instruct for 200B audio & text tokens > Zero-shot voice cloning > Text + audio input/ Audio output > Ultra-fast inference w/ 200ms TTFA > Models on the Hub & Finetuning code on its way! 🚀 What an amazing time to be alive 🤗

Vaibhav (VB) Srivastav

67,003 次观看 • 1 年前

You don't need a GPU to run text-to-speech anymore. Kyutai released Pocket TTS, a 100 million parameter speech model that runs entirely on CPU. How it performs: > Nearly 6x faster than real time on a MacBook CPU > First audio chunk arrives in about 200 milliseconds > Runs on just 2 CPU cores > Supports voice cloning and 6 languages Setup takes one command: pip install pocket-tts Text-to-speech that used to need a GPU or a paid API now runs locally, on a laptop, in near real time.

You don't need a GPU to run text-to-speech anymore. Kyutai released Pocket TTS, a 100 million parameter speech model that runs entirely on CPU. How it performs: > Nearly 6x faster than real time on a MacBook CPU > First audio chunk arrives in about 200 milliseconds > Runs on just 2 CPU cores > Supports voice cloning and 6 languages Setup takes one command: pip install pocket-tts Text-to-speech that used to need a GPU or a paid API now runs locally, on a laptop, in near real time.

Simplifying AI

14,391 次观看 • 1 个月前

Just released on Hugging Face: Vui, a 100M open-source NotebookLM! 3 models: > Vui.BASE is the base checkpoint trained on 40k hours of audio conversations > Vui.ABRAHAM is a single speaker model that can reply with context awareness. > Vui.COHOST is checkpoint with two speakers that can talk to each other. It clones voices, breathes, uhs, [laughs] — even non-speech sounds. Human-like TTS is here!

Just released on Hugging Face: Vui, a 100M open-source NotebookLM! 3 models: > Vui.BASE is the base checkpoint trained on 40k hours of audio conversations > Vui.ABRAHAM is a single speaker model that can reply with context awareness. > Vui.COHOST is checkpoint with two speakers that can talk to each other. It clones voices, breathes, uhs, [laughs] — even non-speech sounds. Human-like TTS is here!

steven

43,661 次观看 • 1 年前

Introducing Soniox TTS v2, our most powerful text-to-speech model yet. Soniox TTS v2 brings extraordinary voice quality, expressive control through audio tags, exceptional precision, high-fidelity voice cloning, more than 60 languages, natural language mixing, and low-latency streaming together in one model. Built from the ground up with a new model architecture, audio codec, and highly optimized inference engine, Soniox TTS v2 delivers frontier TTS at just $0.70 per generated hour. Available globally today as tts-rt-v2. Try it out: Turn the sound on and hear what’s possible. 🔊

Introducing Soniox TTS v2, our most powerful text-to-speech model yet. Soniox TTS v2 brings extraordinary voice quality, expressive control through audio tags, exceptional precision, high-fidelity voice cloning, more than 60 languages, natural language mixing, and low-latency streaming together in one model. Built from the ground up with a new model architecture, audio codec, and highly optimized inference engine, Soniox TTS v2 delivers frontier TTS at just $0.70 per generated hour. Available globally today as tts-rt-v2. Try it out: Turn the sound on and hear what’s possible. 🔊

Soniox

664,741 次观看 • 1 个月前

🔊🔛🔥 ... Groq Inc launched voice generation today. GroqCloud now has realtime transcription, LLMs, *and* text-to-speech. You can build super-responsive, ultra low-latency voice agents end-to-end entirely on Groq! At Daily, we're big fans of Groq's fast, low-latency inference. Pipecat AI supports all the Groq models, including the new voice model. Lately, we've been obsessively playing a voice chat game that Mark Backman wrote. (My high score is 6, so far.) Here's Mark, with Groq's `Celeste-PlayAI` voice.

🔊🔛🔥 ... Groq Inc launched voice generation today. GroqCloud now has realtime transcription, LLMs, and text-to-speech. You can build super-responsive, ultra low-latency voice agents end-to-end entirely on Groq! At Daily, we're big fans of Groq's fast, low-latency inference. Pipecat AI supports all the Groq models, including the new voice model. Lately, we've been obsessively playing a voice chat game that Mark Backman wrote. (My high score is 6, so far.) Here's Mark, with Groq's `Celeste-PlayAI` voice.

kwindla

31,843 次观看 • 1 年前

The kyutai fully end-to-end audio model demo of today is a huge deal that many people missed in the room Mostly irrelevant are the facts that: - they come a few week after OpenAI ChatGPT-4o - the demo was less polished than the 4o one (in terms of voice quality, voice timing…) Relevant: - the model training pipeline and model archi are simple and hugely scalable, with a tiny 8+ people team like Kyutai building it in 4 months. Synthetic data is a huge enabler here - laser focus on local devices: Moshi will soon be everywhere. Frontier model builders have low incentive to let you run smaller models locally (price per token…) but non-profits like Kyutai have very different incentives. The Moshi demo is already online while the OpenAI 4o one is still in limbo. - going under 300 ms of latency while keeping Llama 8B or above quality of answers is a key enabler in terms of interactivity, it’s game changing, This feeling when the model answer your question before you even finished asking is quite crazy or when you interrupt the model while it’s talking and it react… Predictive coding in a model, instantly updated model of what you’re about to say... Basically they nailed the fundamentals. It’s here. This interactive voice tech will be everywhere. It will soon be an obvious commodity.

The kyutai fully end-to-end audio model demo of today is a huge deal that many people missed in the room Mostly irrelevant are the facts that: - they come a few week after OpenAI ChatGPT-4o - the demo was less polished than the 4o one (in terms of voice quality, voice timing…) Relevant: - the model training pipeline and model archi are simple and hugely scalable, with a tiny 8+ people team like Kyutai building it in 4 months. Synthetic data is a huge enabler here - laser focus on local devices: Moshi will soon be everywhere. Frontier model builders have low incentive to let you run smaller models locally (price per token…) but non-profits like Kyutai have very different incentives. The Moshi demo is already online while the OpenAI 4o one is still in limbo. - going under 300 ms of latency while keeping Llama 8B or above quality of answers is a key enabler in terms of interactivity, it’s game changing, This feeling when the model answer your question before you even finished asking is quite crazy or when you interrupt the model while it’s talking and it react… Predictive coding in a model, instantly updated model of what you’re about to say... Basically they nailed the fundamentals. It’s here. This interactive voice tech will be everywhere. It will soon be an obvious commodity.

Thomas Wolf

339,657 次观看 • 2 年前

NEW: Higgs Audio V2 from BosonAI open, unified TTS model w/ voice cloning, beats GPT 4o mini tts and ElevenLabs v2 🔥 > Trained on 10M hours (speech, music, events) > Built on top of Llama 3.2 3B > Works real-time and on edge > Beats GPT-4o-mini-tts, ElevenLabs v2 in prosody & emotion Multi-speaker dialog > Zero-shot voice cloning 🤩 > Available on Hugging Face Kudos to folks at Boson AI for releasing such a brilliant work and all the details around the model! 🤗

NEW: Higgs Audio V2 from BosonAI open, unified TTS model w/ voice cloning, beats GPT 4o mini tts and ElevenLabs v2 🔥 > Trained on 10M hours (speech, music, events) > Built on top of Llama 3.2 3B > Works real-time and on edge > Beats GPT-4o-mini-tts, ElevenLabs v2 in prosody & emotion Multi-speaker dialog > Zero-shot voice cloning 🤩 > Available on Hugging Face Kudos to folks at Boson AI for releasing such a brilliant work and all the details around the model! 🤗

Vaibhav (VB) Srivastav

79,645 次观看 • 1 年前

Excited to introduce Fish Speech 1.4 - now open-source and more powerful than ever! 🎉 Our mission is to make cutting-edge voice tech accessible to everyone. What's new: - Trained on 700k hours of multilingual data (up from 200k) - Now supports 8 languages: English, Chinese, German, Japanese, French, Spanish, Korean, and Arabic - Fully open-source, empowering developers and researchers worldwide Key features: - Lightning-fast TTS with ultra-low latency - Instant voice cloning - Self-host or use our cloud service - Simple, flat-rate pricing Try it out: - Playground: - GitHub: - HuggingFace Model: - Demo: - Product Hunt: We can't wait to see what you'll create with Fish Audio. Happy voice building! 🎧🐠

Excited to introduce Fish Speech 1.4 - now open-source and more powerful than ever! 🎉 Our mission is to make cutting-edge voice tech accessible to everyone. What's new: - Trained on 700k hours of multilingual data (up from 200k) - Now supports 8 languages: English, Chinese, German, Japanese, French, Spanish, Korean, and Arabic - Fully open-source, empowering developers and researchers worldwide Key features: - Lightning-fast TTS with ultra-low latency - Instant voice cloning - Self-host or use our cloud service - Simple, flat-rate pricing Try it out: - Playground: - GitHub: - HuggingFace Model: - Demo: - Product Hunt: We can't wait to see what you'll create with Fish Audio. Happy voice building! 🎧🐠

Fish Audio

149,977 次观看 • 2 年前

JUST IN: Google releases Gemini 1.5, a powerful MoE model. It's a huge breakthrough. The model has the longest context window ever seen: 1 million tokens. It can process 1 hour of video, 11 hours of audio, 30,000 lines of code, or 700,000 words in a single prompt. When tested on text, code, image, audio and video evaluations, 1.5 Pro outperforms 1.0 Pro on 87% of the benchmarks used for developing LLMs. You can can sign up in AI Studio to try it out.

JUST IN: Google releases Gemini 1.5, a powerful MoE model. It's a huge breakthrough. The model has the longest context window ever seen: 1 million tokens. It can process 1 hour of video, 11 hours of audio, 30,000 lines of code, or 700,000 words in a single prompt. When tested on text, code, image, audio and video evaluations, 1.5 Pro outperforms 1.0 Pro on 87% of the benchmarks used for developing LLMs. You can can sign up in AI Studio to try it out.

Lior Alexander

83,432 次观看 • 2 年前

The value of the work we're doing at Optimum is encapsulated quite well by the phrase "speed is money". In modern markets there are real economic advantages to latency reduction. This is nothing new. Wall Street firms have long been optimizing on latency, primarily through colocation and top of the line hardware. However, when it comes to decentralized systems, expensive hardware and geographic concentration are antithetical to their purpose. Therefore we should optimize decentralized network latency through software, which I'm thrilled about because it's exactly what I've spent the better part of the past 2 decades working on with Random Linear Network Coding. Now let’s talk about networking economics, the relationship between speed and money. First, it's important to note that users will only pay for low latency if it can be consistently guaranteed. Second, you can only make that latency guarantee for a certain number of users. This is a universal law of networking. We can model this relationship on a delay curve, shown below. The delay curve is determined by the utilization rate of the network, meaning how much traffic is flowing through the network divided by the network's throughput. As you approach a level of traffic equal to the available throughput, latency trends infinitely higher. On this delay curve we can impose some utility thresholds. These thresholds are the levels of latency which are important to different groups of users because of how that latency guarantee improves their economic outcomes. Finding the point on the curve where each threshold intersects will tell us what level of traffic we can guarantee that level of latency for. Essentially, there exists a finite supply of speed on a network and the highest utility users of that speed are willing to pay more for it. I like to think of this similarly to expedited shipping options on Amazon. This is why we say speed is money, and why we can create a Latency Marketplace. The only way to increase the supply of speed is to fundamentally increase network throughput. This is what we work on at Optimum by using Random Linear Network Coding. The same relationship between traffic and throughput still applies, but now the delay curve is shifted out further to the right. Now more traffic can be processed at the same latency, or the same traffic can be processed at a lower latency. More speed available to the network. More value unlocked for the network’s users. Crucially, that value is no longer only reserved for those who can afford to sit closest to the machine. Expanding the supply of speed widens who can reach each latency threshold, keeping the network's advantage decentralized rather than concentrated in the hands of a few. When nodes join Optimum and participate, they reap the benefits, but they also add to the capacity. Rather than vying against each other in a zero-sum game, nodes help themselves and others.

The value of the work we're doing at Optimum is encapsulated quite well by the phrase "speed is money". In modern markets there are real economic advantages to latency reduction. This is nothing new. Wall Street firms have long been optimizing on latency, primarily through colocation and top of the line hardware. However, when it comes to decentralized systems, expensive hardware and geographic concentration are antithetical to their purpose. Therefore we should optimize decentralized network latency through software, which I'm thrilled about because it's exactly what I've spent the better part of the past 2 decades working on with Random Linear Network Coding. Now let’s talk about networking economics, the relationship between speed and money. First, it's important to note that users will only pay for low latency if it can be consistently guaranteed. Second, you can only make that latency guarantee for a certain number of users. This is a universal law of networking. We can model this relationship on a delay curve, shown below. The delay curve is determined by the utilization rate of the network, meaning how much traffic is flowing through the network divided by the network's throughput. As you approach a level of traffic equal to the available throughput, latency trends infinitely higher. On this delay curve we can impose some utility thresholds. These thresholds are the levels of latency which are important to different groups of users because of how that latency guarantee improves their economic outcomes. Finding the point on the curve where each threshold intersects will tell us what level of traffic we can guarantee that level of latency for. Essentially, there exists a finite supply of speed on a network and the highest utility users of that speed are willing to pay more for it. I like to think of this similarly to expedited shipping options on Amazon. This is why we say speed is money, and why we can create a Latency Marketplace. The only way to increase the supply of speed is to fundamentally increase network throughput. This is what we work on at Optimum by using Random Linear Network Coding. The same relationship between traffic and throughput still applies, but now the delay curve is shifted out further to the right. Now more traffic can be processed at the same latency, or the same traffic can be processed at a lower latency. More speed available to the network. More value unlocked for the network’s users. Crucially, that value is no longer only reserved for those who can afford to sit closest to the machine. Expanding the supply of speed widens who can reach each latency threshold, keeping the network's advantage decentralized rather than concentrated in the hands of a few. When nodes join Optimum and participate, they reap the benefits, but they also add to the capacity. Rather than vying against each other in a zero-sum game, nodes help themselves and others.

Muriel Medard

45,497 次观看 • 2 个月前

Voice agents are awkward, and everyone notices: You ask a question. The agent thinks. You wait. And wait... Nobody wants this. I'd rather talk to a person. If your model's response time is over 300ms, you won't make it. Unfortunately, most text-to-speech models can't get anywhere close to that. I want you to take a look at the latest model released by : TTS-1.5. I built a simple voice agent using the model so you can see it in action and test it on your computer. You'll find the repository link below. The latency numbers of this model are wild: • Max model → under 250ms • Mini model → under 130ms That's 4x faster than prior generations and faster than human response times!

Voice agents are awkward, and everyone notices: You ask a question. The agent thinks. You wait. And wait... Nobody wants this. I'd rather talk to a person. If your model's response time is over 300ms, you won't make it. Unfortunately, most text-to-speech models can't get anywhere close to that. I want you to take a look at the latest model released by : TTS-1.5. I built a simple voice agent using the model so you can see it in action and test it on your computer. You'll find the repository link below. The latency numbers of this model are wild: • Max model → under 250ms • Mini model → under 130ms That's 4x faster than prior generations and faster than human response times!

Santiago

69,456 次观看 • 7 个月前

Mistral just released Voxtral TTS, an open-weight voice model that tries to match ElevenLabs while keeping it free. A 3-billionn param model that fits on a laptop and runs 6X faster than real-time speech. It is split into 3 parts that handle different jobs. - The 3.4B part is the main language brain that understands the text and plans what should be spoken. - The 390M part turns that plan into speech features like rhythm, pronunciation, and tone. - The 300M part converts those speech features into the final compressed audio signal you can play. That split helps because each part only does the job it is best at, which can make the whole system faster and lighter than one giant model doing everything alone. Mistral claims about 90ms to first audio, about 6x faster than real-time speech output, about 3GB RAM after quantization, 9 languages, and voice copying from 5s of audio, even across languages. Its internal tests report 62.8% preference over ElevenLabs Flash on default voices and 69.9% on custom voices, but the bigger point is that companies can keep speech data on their own machines. IMO, enterprise buyers usually care most about control, cost, and latency, So this model should give it great push for voice-based AI apps/products.

Mistral just released Voxtral TTS, an open-weight voice model that tries to match ElevenLabs while keeping it free. A 3-billionn param model that fits on a laptop and runs 6X faster than real-time speech. It is split into 3 parts that handle different jobs. - The 3.4B part is the main language brain that understands the text and plans what should be spoken. - The 390M part turns that plan into speech features like rhythm, pronunciation, and tone. - The 300M part converts those speech features into the final compressed audio signal you can play. That split helps because each part only does the job it is best at, which can make the whole system faster and lighter than one giant model doing everything alone. Mistral claims about 90ms to first audio, about 6x faster than real-time speech output, about 3GB RAM after quantization, 9 languages, and voice copying from 5s of audio, even across languages. Its internal tests report 62.8% preference over ElevenLabs Flash on default voices and 69.9% on custom voices, but the bigger point is that companies can keep speech data on their own machines. IMO, enterprise buyers usually care most about control, cost, and latency, So this model should give it great push for voice-based AI apps/products.

Rohan Paul

15,672 次观看 • 5 个月前

Microsoft did it again! Speech AI models have a major limitation. They slice long recordings into tiny chunks, lose track of who's speaking, and forget all context halfway through. This is exactly what Microsoft's VibeVoice solves. It's an open-source family of frontier voice AI models for both speech recognition and speech generation. Here's what it can do: > VibeVoice-ASR processes up to 60 minutes of audio in a single pass. No chunking. It outputs structured transcriptions with who spoke, when they spoke, and what they said. > You can feed it custom hotwords like names, technical jargon, or domain-specific terms. The model uses them to significantly improve accuracy on specialized content. > VibeVoice-TTS generates up to 90 minutes of multi-speaker speech with up to 4 distinct speakers. Natural turn-taking, emotional expression, all in one pass. > VibeVoice-Realtime is a 0.5B streaming TTS model with ~300ms first-audio latency. Small enough to deploy practically anywhere. All of this is powered by continuous speech tokenizers running at just 7.5 Hz. This ultra-low frame rate preserves audio quality while making long sequences computationally feasible. I have shared the link to the GitHub repo in the replies!

Microsoft did it again! Speech AI models have a major limitation. They slice long recordings into tiny chunks, lose track of who's speaking, and forget all context halfway through. This is exactly what Microsoft's VibeVoice solves. It's an open-source family of frontier voice AI models for both speech recognition and speech generation. Here's what it can do: > VibeVoice-ASR processes up to 60 minutes of audio in a single pass. No chunking. It outputs structured transcriptions with who spoke, when they spoke, and what they said. > You can feed it custom hotwords like names, technical jargon, or domain-specific terms. The model uses them to significantly improve accuracy on specialized content. > VibeVoice-TTS generates up to 90 minutes of multi-speaker speech with up to 4 distinct speakers. Natural turn-taking, emotional expression, all in one pass. > VibeVoice-Realtime is a 0.5B streaming TTS model with ~300ms first-audio latency. Small enough to deploy practically anywhere. All of this is powered by continuous speech tokenizers running at just 7.5 Hz. This ultra-low frame rate preserves audio quality while making long sequences computationally feasible. I have shared the link to the GitHub repo in the replies!

Akshay 🚀

45,256 次观看 • 5 个月前

Introducing PhoneLLM, an open model for voice agents. GPT 5.6 Terra performance on typical voice agent tasks at 1/3 the latency and 1/18 the cost. For voice agents, we need models that are both very low latency and very good at tool calling and instruction following. There's a trade-off here, and we often have to compromise on either latency or capability when building voice agents. With PhoneLLM (and the training and data stack that made this model possible) we're fixing this problem. For the last couple of years, most of the effort in frontier model development has gone towards leveraging test-time compute. Which is awesome! Models of all shapes and sizes are available that perform really, really well ... if you have "thinking" turned on for your model. But if you need your agent to respond at voice conversation speed, you can't use thinking models. PhoneLLM is a full-weights fine-tune of NVIDIA Nemotron Nano 30B. We trained on a wide range of real-world telephone and customer support use cases. The training focused on taking the excellent Nano 30B base capabilities and teaching the model to do typical voice agent tasks with thinking disabled. The results are really good: accurate tool calling and concise, on-topic responses in long conversations. And fast: TTFAT measured server-side is <100ms if you run PhoneLLM on a lightly loaded B200. :-) But seriously, when we characterize model latency, we do it with full, end-to-end, batched request simulations using real Pipecat voice agent pipelines. You can serve more than 80 concurrent agents on a single B200 with P95 end-to-end TTFAT <600ms. Including network overhead. That's an LLM cost-per-minute around $0.0025. (1/4 of a cent.) At a latency lower than any third-party API offers today. More details about this model, including weights on Hugging Face, how to spin it up with one click on Modal, and a starter project repo you can clone, are in the thread ...

Introducing PhoneLLM, an open model for voice agents. GPT 5.6 Terra performance on typical voice agent tasks at 1/3 the latency and 1/18 the cost. For voice agents, we need models that are both very low latency and very good at tool calling and instruction following. There's a trade-off here, and we often have to compromise on either latency or capability when building voice agents. With PhoneLLM (and the training and data stack that made this model possible) we're fixing this problem. For the last couple of years, most of the effort in frontier model development has gone towards leveraging test-time compute. Which is awesome! Models of all shapes and sizes are available that perform really, really well ... if you have "thinking" turned on for your model. But if you need your agent to respond at voice conversation speed, you can't use thinking models. PhoneLLM is a full-weights fine-tune of NVIDIA Nemotron Nano 30B. We trained on a wide range of real-world telephone and customer support use cases. The training focused on taking the excellent Nano 30B base capabilities and teaching the model to do typical voice agent tasks with thinking disabled. The results are really good: accurate tool calling and concise, on-topic responses in long conversations. And fast: TTFAT measured server-side is <100ms if you run PhoneLLM on a lightly loaded B200. :-) But seriously, when we characterize model latency, we do it with full, end-to-end, batched request simulations using real Pipecat voice agent pipelines. You can serve more than 80 concurrent agents on a single B200 with P95 end-to-end TTFAT <600ms. Including network overhead. That's an LLM cost-per-minute around $0.0025. (1/4 of a cent.) At a latency lower than any third-party API offers today. More details about this model, including weights on Hugging Face, how to spin it up with one click on Modal, and a starter project repo you can clone, are in the thread ...

kwindla

330,647 次观看 • 24 天前

Smart Turn v2: open source, native audio turn detection in 14 languages. New checkpoint of the open source, open data, open training code, semantic VAD model on Hugging Face, fal, and Pipecat AI. - 3x faster inference (12ms on an L40) - 14 languages (13 more than v1, which was english-only) - New synthetic data set `chirp_3_all` with ~163k audio samples - 99% accuracy on held out `human_5_all` test data Good turn detection is critical for voice agents. This model "understands" both semantic and audio patterns, and mitigates the voice AI trade-off between unwanted turn latency vs the agent interrupting people before they are finished speaking. Training scripts for both Modal and local training are in the repo. We want to make it as easy as possible to contribute to or customize this model! Here's a demo running the smart-turn model with default settings, aimed at generally hitting 400ms total turn detection time. You can tune things to be faster, too. You can help by contributing data, doing architecture expermints, or cleaning open source data! Keep reading ...

Smart Turn v2: open source, native audio turn detection in 14 languages. New checkpoint of the open source, open data, open training code, semantic VAD model on Hugging Face, fal, and Pipecat AI. - 3x faster inference (12ms on an L40) - 14 languages (13 more than v1, which was english-only) - New synthetic data set `chirp_3_all` with ~163k audio samples - 99% accuracy on held out `human_5_all` test data Good turn detection is critical for voice agents. This model "understands" both semantic and audio patterns, and mitigates the voice AI trade-off between unwanted turn latency vs the agent interrupting people before they are finished speaking. Training scripts for both Modal and local training are in the repo. We want to make it as easy as possible to contribute to or customize this model! Here's a demo running the smart-turn model with default settings, aimed at generally hitting 400ms total turn detection time. You can tune things to be faster, too. You can help by contributing data, doing architecture expermints, or cleaning open source data! Keep reading ...

kwindla

42,246 次观看 • 1 年前

What if your voice AI could interrupt you the moment it figured out your question - sometimes even before you finished asking it? Last week, I sat down with Neil, CEO of Gradium and co-founder of Kyutai , to talk about the future of speech-to-speech models and why he believes today's cascaded voice systems will soon look "archaic and brittle." Some highlights from our conversation: 🎯 How Kyutai built Moshi—a full duplex conversational AI with "negative latency"—in 6 months with just 4-6 people (while big tech teams had 10-20x the resources) 🧠 Why speech-to-speech models lose intelligence compared to their text counterparts (and what's being done about it) 📱 Pocket TTS: The first voice cloning model that runs on your phone's CPU—not GPU, CPU 🤖 Why robotics and spatial audio represent the next frontier (hint: current voice systems completely break in these environments) 👶 The efficiency gap: Babies learn to speak fluently from <5,000 hours of audio. Current models train on millions of hours. We're doing something wrong. My favorite vision from Neil? The first truly contrarian AI that interrupts you mid-sentence to tell you why you're wrong. Not just more natural conversation—but actually useful for testing ideas and playing devil's advocate. Full episode and detailed blog post linked in the comments 👇 What's your take - will speech-to-speech replace cascaded systems, or will modularity keep cascaded architectures dominant even as naturalness improves?

What if your voice AI could interrupt you the moment it figured out your question - sometimes even before you finished asking it? Last week, I sat down with Neil, CEO of Gradium and co-founder of Kyutai , to talk about the future of speech-to-speech models and why he believes today's cascaded voice systems will soon look "archaic and brittle." Some highlights from our conversation: 🎯 How Kyutai built Moshi—a full duplex conversational AI with "negative latency"—in 6 months with just 4-6 people (while big tech teams had 10-20x the resources) 🧠 Why speech-to-speech models lose intelligence compared to their text counterparts (and what's being done about it) 📱 Pocket TTS: The first voice cloning model that runs on your phone's CPU—not GPU, CPU 🤖 Why robotics and spatial audio represent the next frontier (hint: current voice systems completely break in these environments) 👶 The efficiency gap: Babies learn to speak fluently from <5,000 hours of audio. Current models train on millions of hours. We're doing something wrong. My favorite vision from Neil? The first truly contrarian AI that interrupts you mid-sentence to tell you why you're wrong. Not just more natural conversation—but actually useful for testing ideas and playing devil's advocate. Full episode and detailed blog post linked in the comments 👇 What's your take - will speech-to-speech replace cascaded systems, or will modularity keep cascaded architectures dominant even as naturalness improves?

Brooke Hopkins

13,000 次观看 • 7 个月前