Introducing PhoneLLM, an open model for voice agents. GPT 5.6 Terra performance on typical voice agent tasks at 1/3 the latency and 1/18 the cost. For voice agents, we need models that are both very low latency and very good at tool calling and instruction following. There's a trade-off... here, and we often have to compromise on either latency or capability when building voice agents. With PhoneLLM (and the training and data stack that made this model possible) we're fixing this problem. For the last couple of years, most of the effort in frontier model development has gone towards leveraging test-time compute. Which is awesome! Models of all shapes and sizes are available that perform really, really well ... if you have "thinking" turned on for your model. But if you need your agent to respond at voice conversation speed, you can't use thinking models. PhoneLLM is a full-weights fine-tune of NVIDIA Nemotron Nano 30B. We trained on a wide range of real-world telephone and customer support use cases. The training focused on taking the excellent Nano 30B base capabilities and teaching the model to do typical voice agent tasks with thinking disabled. The results are really good: accurate tool calling and concise, on-topic responses in long conversations. And fast: TTFAT measured server-side is <100ms if you run PhoneLLM on a lightly loaded B200. :-) But seriously, when we characterize model latency, we do it with full, end-to-end, batched request simulations using real Pipecat voice agent pipelines. You can serve more than 80 concurrent agents on a single B200 with P95 end-to-end TTFAT <600ms. Including network overhead. That's an LLM cost-per-minute around $0.0025. (1/4 of a cent.) At a latency lower than any third-party API offers today. More details about this model, including weights on Hugging Face, how to spin it up with one click on Modal, and a starter project repo you can clone, are in the thread ...show more

Weights are here:

Ben Shababo at @modal did a bunch of great inference optimization work to achieve the >80 concurrent clients at sub-600ms end-to-end TTFAT. You can spin the model up as a Modal Endpoint with one click or with this command: ``` modal endpoint create --model pipecat-ai/phonellm-alpha-1 ```

Blog post with more details about this model is here: We also have a new internal benchmark, PhoneBench, that's part of this project. And we're working with partners to create custom models using the "factory" that produced PhoneBench. If you need a super low-latency, agentic model for your voice or other realtime AI use case, our training stack might be useful. We can curate and manage production agent traces, generate synthetic data, create RL environments for training multi-turn models, and either run training for you or give you access to our tools. Thank you to the teams at @NVIDIA_AI for the amazing Nemotron family of models and NeMo tooling that we're building on, and @modal for a fantastic platform for both training and inference. More releases coming soon!

yes impressive, voice agent, we are also heading there 🚀

This is great! Just curious, any plan to serve as a public endpoint from ? Hosting a model costs more when it's not been actively used

Yes! We do plan to make the model available pay-as-you-go through Pipecat Cloud. We've mostly worked with bigger companies on this project, who usually want to self-host in their own infrastructure. But we totally understand this request and will work on making it happen.

Make sense! Meanwhile we'll self-host and evaluate the model

Out of curiosity, what's the reason for choosing Nemotron 3 Nano 30B-A3B over Qwen3.6-35B-A3B for the base model?

You can spin up an endpoint on Modal for ~$6/hour with one click in the Modal dashboard.

Now, with double curiosity, was that a bot just replied?

@mark_backman Wow you guys are cooking!

@JonPTaylor Wow. Congrats!

1/18 the cost is huge

@nisten This is such a great work! I can’t wait to use this now! 🔥congrats! voice

Time to cook!!

Sub-600ms end-to-end latency and $0.0025/min just quietly destroyed the business model of like 40 different voice-agent wrapper startups. The open-source catch-up is getting ruthless

I mean, can't the voice agent startups just use PhoneLLM and reduce their costs?

any chance you guys do a nemotron voicelabs 11b (full duplex model) finetune for phone calls? that would be super cool! especially since it can tool call now

I love the VoiceChat and the team working on it is fantastic. I've been tinkering with a DGX Spark port of the VoiceChat inference stack, in my spare time. I do think 11B might be a little too small for general-purpose, reliable tool calling in multi-turn conversations, though, no matter how much we train it. One interesting direction might be to train VoiceChat *never* to call tools, but to know that another model it is paired with will call tools and share output with it. Or, a scaled-up VoiceChat that's the same size (roughly) as Nemotron 3 Nano/Lightning would be amazing. Realtime inference speed for a model of that size is going to be a bit challenging, though. (I think it's possible. But not drop-dead easy!)

thats really cool! will check it out in my experience, the current voicechat is pretty consistent at tool calls even over many turns, is there any specific thing where it fails for you? (like a specific amt of turns?) i'm pretty curious as i've not fully tested it yet

This is the model shape voice agents actually need!

Very useful and amazing solution, good part is that it can serve 80 concurrent calls with B200

Great work, kudos to the Pipecat team! Any plans for releasing PhoneBench as well? 😄

Yes, we'll release PhoneBench. Working on a new benchmarking harness that cleans up and unifies a bunch of our benchmarking work!

huh voice agents might actually work now

1/18 the cost is the number that actually matters for anyone running these at volume

1/18th the cost at 1/3rd the latency is a massive unlock for voice agents.

@ycombinator Feels like a great idea. I love talking to gpt on my phone and in my car. How does the voice actually sound? I’ve noticed gpt sounds very crackly and glitchy

Yeah pipecat! nice work! 🐈

gunna add this to

The killer LLM would be small enough to fit on your iPhone and would be very well trained on tool calling and replacing Siri, because all you need Siri for is to do shit for you.

It feels like we're not that far from useful models running on high-end phones. Increasingly good MoE architectures, distillation/teacher/synthetic-data training techniques, etc. I do think the ~30B/A3B size of Nemotron 3 Nano is kind of the smallest size right now that feels "useful" to me. But that's very subjective.

I don't really care about its reasoning, as long it can do things fast. Anything better than siri

amazing kudos @kwindla for this

niccceee, any tests done on multilingual performance, i am assuming it should be pretty good for that as well

That's a really good question. For this training run, all the new scenario data is English. But the "teaching" here seems to generalize pretty well, so I expect that the base Nemotron 3 Nano multi-lingual capabilities will support multi-lingual for these new use cases the model learns. Would be very interested to hear how it works, if you try it.

lemme try this across a few evals, will probably drop some results over discord

Really cool. Congrats on the launch. You’re a machine! Mad founder respect @kwindla 🫡

This is amazing. Tool calls and complex conversation rules is our priority. Will check it out and let you know

awesome work @kwindla and team

Look forward to hearing any thoughts you have about the PhoneLLM model (or model training in general).

sickkkkkkk!

the latency chart isnt the headline for me. 80 concurrent agents on one B200 is. thats a whole voice product living on a single box instead of a per-minute vendor bill that scales with your success. self-hosting voice finally pencils out

a third the latency is the number that matters for voice, not the benchmark parity. below a certain response time a voice agent stops feeling like a walkie talkie and starts feeling like a conversation

Anything we can do to use it for Spanish voices?!

It should work well in Spanish! Have you tried it?

I will try it, would love to help you any way possible!

The 1/18 cost delta is what actually matters here — at realtime API pricing, heavy tool-loop agents burn through per-minute audio rates fast. Curious how tool call reliability holds up on unseen/shifted schemas though, since in voice a misfired function call costs you a whole conversational turn with no scroll-back to recover. If TTFT stays sub-300ms with proper barge-in and semantic VAD, the hosted realtime endpoints have a real problem on phone workloads.

Have you tested Gemini 3.7 Flash?

Gemini 3.7 Flash is a good model for multi-turn conversation, but the TTFAT is still fairly high. It does not allow reasoning to be turned off. P95 TTFAT is around 1,300ms.

Thanks for the insight!

1/18 the cost and 1/3 the latency. I burned real money on a voice bot that said 'um' more than my callers, so yes, latency and tool calling are the whole game. Frontier models are overkill for 'please reschedule Tuesday'.

@nisten Wow this is awesome! Great work 🚀

This is great! I am a big fan of the benchmarking work. First aiewf and now Phonebench. I am wondering if you are planning a benchmark variant without tool-calling where all the necessary info for the customer is preloaded in a big system prompt.

> I am wondering if you are planning a benchmark variant without tool-calling where all the necessary info for the customer is preloaded in a big system prompt That's an interesting idea. We could definitely build that benchmark. The aiewf-eval `medium-context` benchmark system prompt is quite large. 10k tokens. But that benchmark also does tool calling. Almost all the agents I build use tools. I'd love to hear more about how you structure your agents. What's a typical use case where you don't need to define tools? cc @mark_backman who has been thinking a lot, lately, about cleaning up and extending the aiewf-eval harness work and combining it with other internal benchmarks we have. This would make it easier for anybody using Pipecat to create/maintain benchmarks.

This would be dual-purpose. 1) Test prompt adherence and information retrieval from prompts for models that don't have tool calling. Or test just those capabilities in isolation without a tool API setup. 2) Based on how some users ended up using PersonaPlex without toolcalling...

2)... they would pre-retrieve customer info based on incoming phone number and put that in system prompt. Or in non-specific use cases, pre-retrieve into system prompt info like tech support debug flow/available seats/tax filing steps. Like the Doc2Dial benchmark setup.